[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:15 dist-test-slave-g9sh krb5kdc[28146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:15 dist-test-slave-g9sh krb5kdc[28146](info): set up 2 sockets
Sep 28 16:34:15 dist-test-slave-g9sh krb5kdc[28146](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 16:34:16.711814 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:16 dist-test-slave-g9sh krb5kdc[28146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:34:16.832789 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:41543
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:41543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:16.946643 28159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:16.946873 28159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:16.946911 28159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:16.950749 28159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:16.950791 28159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:16.950810 28159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:16.950826 28159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:16.950841 28159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:16.955950 28159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:41543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:41543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:16.956928 28159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:16.957952 28159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:34:16.967784 28159 server_base.cc:1034] running on GCE node
W20260928 16:34:16.968545 28164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:16.968588 28167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:16.968832 28165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:16.969499 28159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:16.970522 28159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:16.971688 28159 hybrid_clock.cc:648] HybridClock initialized: now 1790613256971650 us; error 47 us; skew 500 ppm
Sep 28 16:34:16 dist-test-slave-g9sh krb5kdc[28146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613256, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:16.974593 28159 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:34:16.975435 28159 webserver.cc:460] Webserver started at http://127.27.122.254:44893/ using document root <none> and password file <none>
I20260928 16:34:16.975983 28159 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:16.976042 28159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:16.976267 28159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:16.978488 28159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "dc37edd8599e4992a5db373d8f2a777e"
format_stamp: "Formatted at 2026-09-28 16:34:16 on dist-test-slave-g9sh"
server_key: "f3bbb2e37f6ad957e9185aa471ef9239"
server_key_iv: "69bf901e63f183cb23df474ec888e954"
server_key_version: "encryptionkey@0"
I20260928 16:34:16.979266 28159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "dc37edd8599e4992a5db373d8f2a777e"
format_stamp: "Formatted at 2026-09-28 16:34:16 on dist-test-slave-g9sh"
server_key: "f3bbb2e37f6ad957e9185aa471ef9239"
server_key_iv: "69bf901e63f183cb23df474ec888e954"
server_key_version: "encryptionkey@0"
I20260928 16:34:16.983219 28159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:34:16.985541 28174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:16.986703 28159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:16.986829 28159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "dc37edd8599e4992a5db373d8f2a777e"
format_stamp: "Formatted at 2026-09-28 16:34:16 on dist-test-slave-g9sh"
server_key: "f3bbb2e37f6ad957e9185aa471ef9239"
server_key_iv: "69bf901e63f183cb23df474ec888e954"
server_key_version: "encryptionkey@0"
I20260928 16:34:16.986928 28159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:17.000708 28159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:17.004055 28159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:17.004266 28159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:17.011567 28159 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:41543
I20260928 16:34:17.011596 28227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:41543 every 8 connection(s)
I20260928 16:34:17.012501 28159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:17.015277 28228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:17.018725 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 28159
I20260928 16:34:17.018904 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:17.020139 28139 external_mini_cluster.cc:1428] Setting key d99198c95540f37dc332708e5bc5b813
I20260928 16:34:17.021206 28228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e: Bootstrap starting.
I20260928 16:34:17.023592 28228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:17.024492 28228 log.cc:824] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:17.026714 28228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e: No bootstrap required, opened a new log
I20260928 16:34:17.031569 28228 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37edd8599e4992a5db373d8f2a777e" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41543 } }
I20260928 16:34:17.031883 28228 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:17.031946 28228 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc37edd8599e4992a5db373d8f2a777e, State: Initialized, Role: FOLLOWER
I20260928 16:34:17.032441 28228 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37edd8599e4992a5db373d8f2a777e" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41543 } }
I20260928 16:34:17.032632 28228 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:17.032711 28228 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:17.032806 28228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:17.033658 28228 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37edd8599e4992a5db373d8f2a777e" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41543 } }
I20260928 16:34:17.033921 28228 leader_election.cc:302] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc37edd8599e4992a5db373d8f2a777e; no voters:
I20260928 16:34:17.034178 28228 leader_election.cc:288] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:17.034233 28233 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:17.034443 28233 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [term 1 LEADER]: Becoming Leader. State: Replica: dc37edd8599e4992a5db373d8f2a777e, State: Running, Role: LEADER
I20260928 16:34:17.034732 28233 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37edd8599e4992a5db373d8f2a777e" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41543 } }
I20260928 16:34:17.035590 28228 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:17.035851 28235 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc37edd8599e4992a5db373d8f2a777e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37edd8599e4992a5db373d8f2a777e" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41543 } } }
I20260928 16:34:17.035982 28235 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:17.036248 28234 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc37edd8599e4992a5db373d8f2a777e. Latest consensus state: current_term: 1 leader_uuid: "dc37edd8599e4992a5db373d8f2a777e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc37edd8599e4992a5db373d8f2a777e" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41543 } } }
I20260928 16:34:17.036338 28234 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:17.036414 28237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:17.039784 28237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:17.045825 28237 catalog_manager.cc:1347] Generated new cluster ID: 49381b6fe2f64052960356c6a4df95a9
I20260928 16:34:17.045890 28237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:17.049271 28230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:17.023176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.27.122.254:41543)
0928 16:34:17.023658 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:17.023677 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:17.024902 (+ 1225us) server_negotiation.cc:407] Connection header received
0928 16:34:17.029221 (+ 4319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:17.029239 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:17.029522 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:17.029777 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:17.032979 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.034294 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.036273 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.037096 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.037324 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:17.042782 (+ 5458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:17.042789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:17.042804 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:17.042846 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:17.045280 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:17.045829 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:17.045848 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:17.045852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:17.046543 (+ 691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:17.046939 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:17.046943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:17.046944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:17.047324 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:17.047569 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:17.048126 (+ 557us) server_negotiation.cc:299] Negotiation successful
0928 16:34:17.048432 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":143,"threads_started":1}
I20260928 16:34:17.070271 28237 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:17.071409 28237 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:17.078383 28237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc37edd8599e4992a5db373d8f2a777e: Generated new TSK 0
I20260928 16:34:17.079012 28237 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:34:17.110325 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41543
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:17.231851 28254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:17.232091 28254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:17.232134 28254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:17.236289 28254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:17.236373 28254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:17.236487 28254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:17.242013 28254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:17.243112 28254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:17.244186 28254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:17.253845 28259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:17.253842 28260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:17.253842 28262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:17.254182 28254 server_base.cc:1034] running on GCE node
I20260928 16:34:17.254580 28254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:17.255191 28254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:17.256376 28254 hybrid_clock.cc:648] HybridClock initialized: now 1790613257256365 us; error 28 us; skew 500 ppm
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613257, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:17.259127 28254 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:34:17.259953 28254 webserver.cc:460] Webserver started at http://127.27.122.193:32781/ using document root <none> and password file <none>
I20260928 16:34:17.260456 28254 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:17.260514 28254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:17.260684 28254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:17.262761 28254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "418007b9e9f34252a6b77cf9947aeeb7"
format_stamp: "Formatted at 2026-09-28 16:34:17 on dist-test-slave-g9sh"
server_key: "5bfd1e3a50218be75dcb9c221ed1d682"
server_key_iv: "ca3e5e8d538554193d19250542faf9cc"
server_key_version: "encryptionkey@0"
I20260928 16:34:17.263324 28254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "418007b9e9f34252a6b77cf9947aeeb7"
format_stamp: "Formatted at 2026-09-28 16:34:17 on dist-test-slave-g9sh"
server_key: "5bfd1e3a50218be75dcb9c221ed1d682"
server_key_iv: "ca3e5e8d538554193d19250542faf9cc"
server_key_version: "encryptionkey@0"
I20260928 16:34:17.267102 28254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:34:17.269577 28269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:17.270804 28254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:34:17.270896 28254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "418007b9e9f34252a6b77cf9947aeeb7"
format_stamp: "Formatted at 2026-09-28 16:34:17 on dist-test-slave-g9sh"
server_key: "5bfd1e3a50218be75dcb9c221ed1d682"
server_key_iv: "ca3e5e8d538554193d19250542faf9cc"
server_key_version: "encryptionkey@0"
I20260928 16:34:17.270980 28254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:17.299835 28254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:17.302891 28254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:17.303108 28254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:17.303815 28254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:17.304716 28254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:17.304765 28254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:17.304806 28254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:17.304828 28254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:17.315623 28254 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:37871
I20260928 16:34:17.315646 28382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:37871 every 8 connection(s)
I20260928 16:34:17.316529 28254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:17.316787 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 28254
I20260928 16:34:17.316937 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:34:17.317212 28139 external_mini_cluster.cc:1428] Setting key 71d734107a0ba1cd77e1b60834fbfca8
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613257, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:17.335542 28230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:17.318530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:33679 (local address 127.27.122.254:41543)
0928 16:34:17.318685 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:17.318691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:17.319788 (+ 1097us) server_negotiation.cc:407] Connection header received
0928 16:34:17.321243 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:17.321247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:17.321306 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:17.321401 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:17.322570 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.323117 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.324274 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.324668 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.324779 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:17.330725 (+ 5946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:17.330731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:17.330736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:17.330769 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:17.332863 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:17.333566 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:17.333570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:17.333572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:17.333666 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:17.334035 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:17.334038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:17.334040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:17.334295 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:17.334394 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:17.335091 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 16:34:17.335250 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:34:17.336282 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:17.319064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41543 (local address 127.27.122.193:33679)
0928 16:34:17.319620 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:17.319673 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:17.321035 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:17.321554 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:17.321571 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:17.321937 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:17.322413 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:17.322426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.323249 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.323253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:17.324140 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:17.324150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.324992 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.324999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:17.325248 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:17.326001 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:17.326025 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:17.330523 (+ 4498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:17.333045 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:17.333052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:17.333068 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:17.333452 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:17.333766 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:17.333769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:17.333771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:17.333923 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:17.334453 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:17.334460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:17.334805 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 16:34:17.334995 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 16:34:17.335321 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":198,"threads_started":1}
I20260928 16:34:17.337802 28383 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41543
I20260928 16:34:17.338019 28383 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:17.338469 28383 heartbeater.cc:511] Master 127.27.122.254:41543 requested a full tablet report, sending...
I20260928 16:34:17.340101 28191 ts_manager.cc:194] Registered new tserver with Master: 418007b9e9f34252a6b77cf9947aeeb7 (127.27.122.193:37871)
I20260928 16:34:17.341481 28191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:33679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:34:17.345969 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41543
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:17.458806 28388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:17.459056 28388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:17.459120 28388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:17.463452 28388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:17.463501 28388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:17.463557 28388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:17.468628 28388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:17.469527 28388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:17.470413 28388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:34:17.480238 28388 server_base.cc:1034] running on GCE node
W20260928 16:34:17.480213 28396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:17.480188 28393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:17.480182 28394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:17.480855 28388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:17.481473 28388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:17.482654 28388 hybrid_clock.cc:648] HybridClock initialized: now 1790613257482627 us; error 40 us; skew 500 ppm
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613257, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:17.485374 28388 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:34:17.486243 28388 webserver.cc:460] Webserver started at http://127.27.122.194:34031/ using document root <none> and password file <none>
I20260928 16:34:17.486783 28388 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:17.486873 28388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:17.487107 28388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:17.489262 28388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "65a4f3d4262749169a75c1172fd732ba"
format_stamp: "Formatted at 2026-09-28 16:34:17 on dist-test-slave-g9sh"
server_key: "6dbd5d4c6e0d4ab80d12133f8bdea3db"
server_key_iv: "c51c0c1f6a61ff0fe59749e75a149a18"
server_key_version: "encryptionkey@0"
I20260928 16:34:17.489840 28388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "65a4f3d4262749169a75c1172fd732ba"
format_stamp: "Formatted at 2026-09-28 16:34:17 on dist-test-slave-g9sh"
server_key: "6dbd5d4c6e0d4ab80d12133f8bdea3db"
server_key_iv: "c51c0c1f6a61ff0fe59749e75a149a18"
server_key_version: "encryptionkey@0"
I20260928 16:34:17.493531 28388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:34:17.495787 28403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:17.497030 28388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:17.497161 28388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "65a4f3d4262749169a75c1172fd732ba"
format_stamp: "Formatted at 2026-09-28 16:34:17 on dist-test-slave-g9sh"
server_key: "6dbd5d4c6e0d4ab80d12133f8bdea3db"
server_key_iv: "c51c0c1f6a61ff0fe59749e75a149a18"
server_key_version: "encryptionkey@0"
I20260928 16:34:17.497272 28388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:17.520701 28388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:17.524116 28388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:17.524328 28388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:17.524995 28388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:17.525856 28388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:17.525929 28388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:17.526002 28388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:17.526046 28388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:17.536007 28388 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:34445
I20260928 16:34:17.536139 28516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:34445 every 8 connection(s)
I20260928 16:34:17.537098 28388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:17.541756 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 28388
I20260928 16:34:17.541851 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:34:17.542140 28139 external_mini_cluster.cc:1428] Setting key 479777664427609227383915a1f489f1
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613257, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:17.551963 28230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:17.538874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:56979 (local address 127.27.122.254:41543)
0928 16:34:17.539004 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:17.539008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:17.539832 (+ 824us) server_negotiation.cc:407] Connection header received
0928 16:34:17.541133 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:17.541136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:17.541185 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:17.541260 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:17.542626 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.543280 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.544495 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.544936 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.545052 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:17.547870 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:17.547873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:17.547875 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:17.547899 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:17.549621 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:17.550248 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:17.550251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:17.550252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:17.550360 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:17.550826 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:17.550828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:17.550829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:17.550987 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:17.551069 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:17.551755 (+ 686us) server_negotiation.cc:299] Negotiation successful
0928 16:34:17.551849 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
I20260928 16:34:17.553125 28519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:17.539230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41543 (local address 127.27.122.194:56979)
0928 16:34:17.539678 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:17.539719 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:17.540876 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:17.541422 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:17.541429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:17.541843 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:17.542442 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:17.542458 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.543415 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.543418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:17.544368 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:17.544378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.545128 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.545134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:17.545364 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:17.545983 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:17.546007 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:17.547679 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:17.549760 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:17.549769 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:17.549784 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:17.550145 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:17.550458 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:17.550463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:17.550466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:17.550723 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:17.551100 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:17.551106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:17.551463 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 16:34:17.551721 (+ 258us) client_negotiation.cc:240] Negotiation successful
0928 16:34:17.552085 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260928 16:34:17.554201 28517 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41543
I20260928 16:34:17.554435 28517 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:17.554924 28517 heartbeater.cc:511] Master 127.27.122.254:41543 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
I20260928 16:34:17.556046 28191 ts_manager.cc:194] Registered new tserver with Master: 65a4f3d4262749169a75c1172fd732ba (127.27.122.194:34445)
I20260928 16:34:17.556849 28191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:56979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:34:17.571345 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41543
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:46033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:17.687191 28522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:17.687434 28522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:17.687526 28522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:17.691587 28522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:17.691644 28522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:17.691767 28522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:17.696789 28522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:46033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:17.697772 28522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:17.698784 28522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:34:17.708601 28522 server_base.cc:1034] running on GCE node
W20260928 16:34:17.708621 28528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:17.708674 28530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:17.708593 28527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:17.709275 28522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:17.709818 28522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:17.710978 28522 hybrid_clock.cc:648] HybridClock initialized: now 1790613257710939 us; error 50 us; skew 500 ppm
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613257, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:17.713636 28522 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:34:17.714481 28522 webserver.cc:460] Webserver started at http://127.27.122.195:46175/ using document root <none> and password file <none>
I20260928 16:34:17.715019 28522 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:17.715107 28522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:17.715344 28522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:17.717415 28522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "7e84ae32ed5942e88bf90513efde53de"
format_stamp: "Formatted at 2026-09-28 16:34:17 on dist-test-slave-g9sh"
server_key: "5d2391d61f02493ec52984f3570d552f"
server_key_iv: "b9e7fc40a83295cd0846c29ee78925cb"
server_key_version: "encryptionkey@0"
I20260928 16:34:17.717971 28522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "7e84ae32ed5942e88bf90513efde53de"
format_stamp: "Formatted at 2026-09-28 16:34:17 on dist-test-slave-g9sh"
server_key: "5d2391d61f02493ec52984f3570d552f"
server_key_iv: "b9e7fc40a83295cd0846c29ee78925cb"
server_key_version: "encryptionkey@0"
I20260928 16:34:17.721544 28522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:34:17.723803 28537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:17.725099 28522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 16:34:17.725231 28522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "7e84ae32ed5942e88bf90513efde53de"
format_stamp: "Formatted at 2026-09-28 16:34:17 on dist-test-slave-g9sh"
server_key: "5d2391d61f02493ec52984f3570d552f"
server_key_iv: "b9e7fc40a83295cd0846c29ee78925cb"
server_key_version: "encryptionkey@0"
I20260928 16:34:17.725342 28522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:17.738896 28522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:17.741549 28522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:17.741752 28522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:17.742300 28522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:17.743080 28522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:17.743153 28522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:17.743220 28522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:17.743270 28522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:17.751807 28522 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:41837
I20260928 16:34:17.751840 28650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:41837 every 8 connection(s)
I20260928 16:34:17.752636 28522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:34:17.756871 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 28522
I20260928 16:34:17.757017 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:34:17.757292 28139 external_mini_cluster.cc:1428] Setting key 7709bbfc35286314ef03aed97d277f05
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613257, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:17.765931 28230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:17.754299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:33179 (local address 127.27.122.254:41543)
0928 16:34:17.754428 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:17.754432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:17.755146 (+ 714us) server_negotiation.cc:407] Connection header received
0928 16:34:17.756435 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:17.756438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:17.756502 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:17.756653 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:17.757956 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.758485 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.759630 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.759855 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.759966 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:17.762390 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:17.762396 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:17.762404 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:17.762427 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:17.763798 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:17.764298 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:17.764303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:17.764305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:17.764388 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:17.764731 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:17.764734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:17.764736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:17.764966 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:17.765077 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:17.765632 (+ 555us) server_negotiation.cc:299] Negotiation successful
0928 16:34:17.765761 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:34:17.766634 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:17.754572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41543 (local address 127.27.122.195:33179)
0928 16:34:17.754994 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:17.755034 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:17.756223 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:17.756825 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:17.756831 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:17.757303 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:17.757805 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:17.757819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.758618 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.758622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:17.759495 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:17.759505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.760001 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.760006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:17.760189 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:17.760767 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:17.760784 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:17.762233 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:17.763912 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:17.763918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:17.763931 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:17.764191 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:17.764483 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:17.764486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:17.764489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:17.764623 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:17.765095 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:17.765101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:17.765362 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 16:34:17.765583 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 16:34:17.765890 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":115,"threads_started":1}
I20260928 16:34:17.767640 28651 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41543
I20260928 16:34:17.767834 28651 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:17.768234 28651 heartbeater.cc:511] Master 127.27.122.254:41543 requested a full tablet report, sending...
I20260928 16:34:17.769162 28191 ts_manager.cc:194] Registered new tserver with Master: 7e84ae32ed5942e88bf90513efde53de (127.27.122.195:41837)
I20260928 16:34:17.769722 28191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:33179
I20260928 16:34:17.771430 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613257, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613257, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.122.193@KRBTEST.COM
I20260928 16:34:17.796291 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:17.787392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.27.122.193:37871)
0928 16:34:17.787682 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:17.787695 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:17.787718 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:34:17.787829 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:17.787834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:17.788006 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:17.788135 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:17.788755 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.789586 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.790446 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.790827 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.790910 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:17.793254 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:17.793264 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:17.793275 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:17.793300 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:17.794691 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:17.795072 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:17.795085 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:17.795086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:17.795166 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:17.795395 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:17.795398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:17.795399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:17.795617 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:17.795760 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:17.795937 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 16:34:17.796116 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
W20260928 16:34:17.797302 28297 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39688
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.193@KRBTEST.COM
I20260928 16:34:17.821381 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:17.813946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39702 (local address 127.27.122.193:37871)
0928 16:34:17.814101 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:17.814104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:17.814172 (+ 68us) server_negotiation.cc:407] Connection header received
0928 16:34:17.814282 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:17.814284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:17.814321 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:17.814395 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:17.815039 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.815529 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.816282 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:17.816494 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:17.816568 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:17.818736 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:17.818739 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:17.818741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:17.818762 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:17.819950 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:17.820307 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:17.820310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:17.820311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:17.820379 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:17.820598 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:17.820600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:17.820601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:17.820737 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:17.820830 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:17.821094 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 16:34:17.821182 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:34:17.822310 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 28254
I20260928 16:34:17.829164 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 28388
I20260928 16:34:17.836935 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 28522
I20260928 16:34:17.842628 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 28159
2026-09-28T16:34:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2196 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28665](info): set up 2 sockets
Sep 28 16:34:17 dist-test-slave-g9sh krb5kdc[28665](info): commencing operation
krb5kdc: starting...
W20260928 16:34:18.887042 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:18 dist-test-slave-g9sh krb5kdc[28665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:34:19.005532 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:46103
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:46103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:19.112927 28678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:19.113231 28678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:19.113304 28678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:19.117264 28678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:19.117355 28678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:19.117393 28678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:19.117431 28678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:19.117461 28678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:19.122617 28678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:46103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:46103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:19.123656 28678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:19.124681 28678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:19.133278 28684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:19.133356 28686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:19.133322 28678 server_base.cc:1034] running on GCE node
W20260928 16:34:19.133306 28683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:19.134055 28678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:19.135138 28678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:19.136327 28678 hybrid_clock.cc:648] HybridClock initialized: now 1790613259136297 us; error 39 us; skew 500 ppm
Sep 28 16:34:19 dist-test-slave-g9sh krb5kdc[28665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613259, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:19.139041 28678 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:34:19.139955 28678 webserver.cc:460] Webserver started at http://127.27.122.254:35933/ using document root <none> and password file <none>
I20260928 16:34:19.140523 28678 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:19.140591 28678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:19.140756 28678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:19.142791 28678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "b489ff0affd946d3b39b19b618de0a81"
format_stamp: "Formatted at 2026-09-28 16:34:19 on dist-test-slave-g9sh"
server_key: "6a45236901267e981781f41eda8586c5"
server_key_iv: "9534c5f4fed08cfd021a911771d9e8bd"
server_key_version: "encryptionkey@0"
I20260928 16:34:19.143316 28678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "b489ff0affd946d3b39b19b618de0a81"
format_stamp: "Formatted at 2026-09-28 16:34:19 on dist-test-slave-g9sh"
server_key: "6a45236901267e981781f41eda8586c5"
server_key_iv: "9534c5f4fed08cfd021a911771d9e8bd"
server_key_version: "encryptionkey@0"
I20260928 16:34:19.147006 28678 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:34:19.149389 28693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:19.150686 28678 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:19.150787 28678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "b489ff0affd946d3b39b19b618de0a81"
format_stamp: "Formatted at 2026-09-28 16:34:19 on dist-test-slave-g9sh"
server_key: "6a45236901267e981781f41eda8586c5"
server_key_iv: "9534c5f4fed08cfd021a911771d9e8bd"
server_key_version: "encryptionkey@0"
I20260928 16:34:19.150875 28678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:19.162465 28678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:19.165105 28678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:19.165244 28678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:19.172196 28678 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:46103
I20260928 16:34:19.172227 28745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:46103 every 8 connection(s)
I20260928 16:34:19.173133 28678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:19.175746 28746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:19.180940 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 28678
I20260928 16:34:19.181062 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:19.181267 28746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81: Bootstrap starting.
I20260928 16:34:19.181324 28139 external_mini_cluster.cc:1428] Setting key 406f09432b0c54b23dabde34f0afacef
I20260928 16:34:19.183702 28746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:19.184425 28746 log.cc:824] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:19.186352 28746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81: No bootstrap required, opened a new log
Sep 28 16:34:19 dist-test-slave-g9sh krb5kdc[28665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:19.190049 28746 raft_consensus.cc:348] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b489ff0affd946d3b39b19b618de0a81" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46103 } }
I20260928 16:34:19.190290 28746 raft_consensus.cc:374] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:19.190387 28746 raft_consensus.cc:729] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b489ff0affd946d3b39b19b618de0a81, State: Initialized, Role: FOLLOWER
I20260928 16:34:19.190858 28746 consensus_queue.cc:260] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b489ff0affd946d3b39b19b618de0a81" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46103 } }
I20260928 16:34:19.191075 28746 raft_consensus.cc:388] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:19.191171 28746 raft_consensus.cc:482] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:19.191303 28746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:19.192152 28746 raft_consensus.cc:504] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b489ff0affd946d3b39b19b618de0a81" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46103 } }
I20260928 16:34:19.192492 28746 leader_election.cc:302] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b489ff0affd946d3b39b19b618de0a81; no voters:
I20260928 16:34:19.192821 28746 leader_election.cc:288] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:19.192885 28751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:19.193153 28751 raft_consensus.cc:686] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [term 1 LEADER]: Becoming Leader. State: Replica: b489ff0affd946d3b39b19b618de0a81, State: Running, Role: LEADER
I20260928 16:34:19.193497 28751 consensus_queue.cc:237] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b489ff0affd946d3b39b19b618de0a81" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46103 } }
I20260928 16:34:19.194167 28746 sys_catalog.cc:564] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:19.195111 28752 sys_catalog.cc:455] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b489ff0affd946d3b39b19b618de0a81. Latest consensus state: current_term: 1 leader_uuid: "b489ff0affd946d3b39b19b618de0a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b489ff0affd946d3b39b19b618de0a81" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46103 } } }
I20260928 16:34:19.195233 28752 sys_catalog.cc:458] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:19.195736 28760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:19.195740 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.182663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.27.122.254:46103)
0928 16:34:19.183068 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:19.183083 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:19.183117 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:34:19.183731 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:19.183747 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:19.184119 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:19.184446 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:19.185271 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.186233 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.187171 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.187650 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.187828 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.190353 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:19.190357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:19.190370 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:19.190397 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:19.192410 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:19.192928 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:19.192936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:19.192942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:19.193091 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:19.193434 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:19.193445 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:19.193447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:19.193898 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:19.194116 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:19.194408 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 16:34:19.194745 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
I20260928 16:34:19.195895 28753 sys_catalog.cc:455] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b489ff0affd946d3b39b19b618de0a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b489ff0affd946d3b39b19b618de0a81" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46103 } } }
I20260928 16:34:19.195968 28753 sys_catalog.cc:458] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:19.199939 28760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:19.205694 28760 catalog_manager.cc:1347] Generated new cluster ID: 785a3e25f48c4842aa286df7dca89759
I20260928 16:34:19.205763 28760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:19.222132 28760 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:19.223331 28760 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:19.229359 28760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b489ff0affd946d3b39b19b618de0a81: Generated new TSK 0
I20260928 16:34:19.230185 28760 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:34:19.260597 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:46103
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:19.374081 28772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:19.374320 28772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:19.374389 28772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:19.378415 28772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:19.378465 28772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:19.378559 28772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:19.383641 28772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:46103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:19.384622 28772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:19.385694 28772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:19.395212 28777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:19.395239 28780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:19.395362 28772 server_base.cc:1034] running on GCE node
W20260928 16:34:19.395212 28778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:19.396054 28772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:19.396798 28772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:19.398017 28772 hybrid_clock.cc:648] HybridClock initialized: now 1790613259397998 us; error 29 us; skew 500 ppm
Sep 28 16:34:19 dist-test-slave-g9sh krb5kdc[28665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613259, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:19.400740 28772 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:34:19.401615 28772 webserver.cc:460] Webserver started at http://127.27.122.193:37825/ using document root <none> and password file <none>
I20260928 16:34:19.402168 28772 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:19.402261 28772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:19.402486 28772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:19.404450 28772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "eb27eb12f5c74bf591c1ff559119bca1"
format_stamp: "Formatted at 2026-09-28 16:34:19 on dist-test-slave-g9sh"
server_key: "dc05ec7ef47cd91205991fb544243ceb"
server_key_iv: "a90002e13090964566c63bf4abea9978"
server_key_version: "encryptionkey@0"
I20260928 16:34:19.405067 28772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "eb27eb12f5c74bf591c1ff559119bca1"
format_stamp: "Formatted at 2026-09-28 16:34:19 on dist-test-slave-g9sh"
server_key: "dc05ec7ef47cd91205991fb544243ceb"
server_key_iv: "a90002e13090964566c63bf4abea9978"
server_key_version: "encryptionkey@0"
I20260928 16:34:19.408694 28772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:34:19.411023 28787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:19.412209 28772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:19.412339 28772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "eb27eb12f5c74bf591c1ff559119bca1"
format_stamp: "Formatted at 2026-09-28 16:34:19 on dist-test-slave-g9sh"
server_key: "dc05ec7ef47cd91205991fb544243ceb"
server_key_iv: "a90002e13090964566c63bf4abea9978"
server_key_version: "encryptionkey@0"
I20260928 16:34:19.412443 28772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:19.428232 28772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:19.430948 28772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:19.431150 28772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:19.431713 28772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:19.432495 28772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:19.432574 28772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:19.432647 28772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:19.432701 28772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:19.441067 28772 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:45751
I20260928 16:34:19.441115 28900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:45751 every 8 connection(s)
I20260928 16:34:19.441907 28772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:19.445953 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 28772
I20260928 16:34:19.446046 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:34:19.446244 28139 external_mini_cluster.cc:1428] Setting key f62fc654de56f3382fb3359f6e0e16c1
Sep 28 16:34:19 dist-test-slave-g9sh krb5kdc[28665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613259, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:34:19.455598 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.443601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:53887 (local address 127.27.122.254:46103)
0928 16:34:19.443733 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:19.443737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:19.444465 (+ 728us) server_negotiation.cc:407] Connection header received
0928 16:34:19.446089 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:19.446093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:19.446155 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:19.446263 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:19.447562 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.448088 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.449297 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.449539 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.449654 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.452055 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:19.452071 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:19.452073 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:19.452096 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:19.453554 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:19.454104 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:19.454107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:19.454109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:19.454183 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:19.454524 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:19.454529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:19.454531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:19.454728 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:19.454825 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:19.455313 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 16:34:19.455429 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:34:19.456389 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.443890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:46103 (local address 127.27.122.193:53887)
0928 16:34:19.444310 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:19.444351 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:19.445870 (+ 1519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:19.446446 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:19.446453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:19.446788 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:19.447405 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:19.447426 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.448220 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.448223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:19.449173 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:19.449184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.449689 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.449693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:19.449860 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.450446 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:19.450462 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:19.451893 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:19.453701 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:19.453707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:19.453721 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:19.453978 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:19.454284 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:19.454287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:19.454289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:19.454426 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:19.454824 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:19.454831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:19.455063 (+ 232us) client_negotiation.cc:769] Sending connection context
0928 16:34:19.455247 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 16:34:19.455547 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
I20260928 16:34:19.457623 28901 heartbeater.cc:347] Connected to a master server at 127.27.122.254:46103
I20260928 16:34:19.457835 28901 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:19.458300 28901 heartbeater.cc:511] Master 127.27.122.254:46103 requested a full tablet report, sending...
I20260928 16:34:19.459733 28710 ts_manager.cc:194] Registered new tserver with Master: eb27eb12f5c74bf591c1ff559119bca1 (127.27.122.193:45751)
I20260928 16:34:19.460952 28710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:53887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:34:19.472152 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:46103
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:19.581127 28906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:19.581394 28906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:19.581467 28906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:19.585474 28906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:19.585546 28906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:19.585633 28906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:19.590700 28906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:46103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:19.591668 28906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:19.592633 28906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:34:19.602131 28906 server_base.cc:1034] running on GCE node
W20260928 16:34:19.602087 28911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:19.602072 28912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:19.602079 28914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:19.602741 28906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:19.603317 28906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:19.604502 28906 hybrid_clock.cc:648] HybridClock initialized: now 1790613259604487 us; error 26 us; skew 500 ppm
Sep 28 16:34:19 dist-test-slave-g9sh krb5kdc[28665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613259, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:19.607237 28906 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:34:19.608018 28906 webserver.cc:460] Webserver started at http://127.27.122.194:45213/ using document root <none> and password file <none>
I20260928 16:34:19.608558 28906 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:19.608649 28906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:19.608870 28906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:19.610816 28906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "399f5ba0bab24b0db563ba825ed66dcd"
format_stamp: "Formatted at 2026-09-28 16:34:19 on dist-test-slave-g9sh"
server_key: "0788bfd12ae907b83d81e8336f182376"
server_key_iv: "f5b5ea3625199befad22451655acaec6"
server_key_version: "encryptionkey@0"
I20260928 16:34:19.611358 28906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "399f5ba0bab24b0db563ba825ed66dcd"
format_stamp: "Formatted at 2026-09-28 16:34:19 on dist-test-slave-g9sh"
server_key: "0788bfd12ae907b83d81e8336f182376"
server_key_iv: "f5b5ea3625199befad22451655acaec6"
server_key_version: "encryptionkey@0"
I20260928 16:34:19.614815 28906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 16:34:19.617322 28921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:19.618461 28906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:19.618592 28906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "399f5ba0bab24b0db563ba825ed66dcd"
format_stamp: "Formatted at 2026-09-28 16:34:19 on dist-test-slave-g9sh"
server_key: "0788bfd12ae907b83d81e8336f182376"
server_key_iv: "f5b5ea3625199befad22451655acaec6"
server_key_version: "encryptionkey@0"
I20260928 16:34:19.618696 28906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:19.629091 28906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:19.632117 28906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:19.632303 28906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:19.632829 28906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:19.633641 28906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:19.633711 28906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:19.633777 28906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:19.633829 28906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:19.642210 28906 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:36271
I20260928 16:34:19.642246 29034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:36271 every 8 connection(s)
I20260928 16:34:19.643015 28906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:19.647559 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 28906
I20260928 16:34:19.647697 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:34:19.647976 28139 external_mini_cluster.cc:1428] Setting key 2da295fb00c32d9217abc2194532095c
Sep 28 16:34:19 dist-test-slave-g9sh krb5kdc[28665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613259, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
I20260928 16:34:19.657687 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.644704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:47441 (local address 127.27.122.254:46103)
0928 16:34:19.644869 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:19.644874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:19.645672 (+ 798us) server_negotiation.cc:407] Connection header received
0928 16:34:19.646856 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:19.646858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:19.646906 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:19.646981 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:19.648467 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.649222 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.650961 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.651295 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.651410 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.653909 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:19.653913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:19.653916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:19.653950 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:19.655616 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:19.656134 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:19.656139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:19.656140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:19.656222 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:19.656561 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:19.656566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:19.656568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:19.656750 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:19.656857 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:19.657391 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 16:34:19.657511 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 16:34:19.658525 29037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.645102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:46103 (local address 127.27.122.194:47441)
0928 16:34:19.645529 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:19.645570 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:19.646664 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:19.647149 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:19.647155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:19.647634 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:19.648299 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:19.648324 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.649387 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.649390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:19.650829 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:19.650845 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.651446 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.651449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:19.651597 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.652173 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:19.652189 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:19.653728 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:19.655733 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:19.655739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:19.655752 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:19.656020 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:19.656318 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:19.656321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:19.656323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:19.656463 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:19.656868 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:19.656876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:19.657141 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 16:34:19.657331 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 16:34:19.657633 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":102,"threads_started":1}
I20260928 16:34:19.659672 29035 heartbeater.cc:347] Connected to a master server at 127.27.122.254:46103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
I20260928 16:34:19.659933 29035 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:19.660516 29035 heartbeater.cc:511] Master 127.27.122.254:46103 requested a full tablet report, sending...
I20260928 16:34:19.661798 28710 ts_manager.cc:194] Registered new tserver with Master: 399f5ba0bab24b0db563ba825ed66dcd (127.27.122.194:36271)
I20260928 16:34:19.662613 28710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:47441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:34:19.675384 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:46103
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:40141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:19.783032 29040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:19.783263 29040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:19.783304 29040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:19.787314 29040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:19.787364 29040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:19.787449 29040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:19.792646 29040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:40141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:46103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:19.793638 29040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:19.794646 29040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:19.804037 29046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:19.804117 29048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:19.804068 29045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:19.804196 29040 server_base.cc:1034] running on GCE node
I20260928 16:34:19.804698 29040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:19.805354 29040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:19.806540 29040 hybrid_clock.cc:648] HybridClock initialized: now 1790613259806524 us; error 24 us; skew 500 ppm
Sep 28 16:34:19 dist-test-slave-g9sh krb5kdc[28665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613259, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:19.809178 29040 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:34:19.810086 29040 webserver.cc:460] Webserver started at http://127.27.122.195:42121/ using document root <none> and password file <none>
I20260928 16:34:19.810621 29040 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:19.810707 29040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:19.810916 29040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:19.812846 29040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "c0f1993cae4e40f1944832fe7548ad4f"
format_stamp: "Formatted at 2026-09-28 16:34:19 on dist-test-slave-g9sh"
server_key: "2e1962ff5abd655505a49f0569f771c8"
server_key_iv: "f98632def9d4aecdecca10fa9cbbd9b6"
server_key_version: "encryptionkey@0"
I20260928 16:34:19.813468 29040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "c0f1993cae4e40f1944832fe7548ad4f"
format_stamp: "Formatted at 2026-09-28 16:34:19 on dist-test-slave-g9sh"
server_key: "2e1962ff5abd655505a49f0569f771c8"
server_key_iv: "f98632def9d4aecdecca10fa9cbbd9b6"
server_key_version: "encryptionkey@0"
I20260928 16:34:19.817121 29040 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:34:19.819404 29055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:19.820640 29040 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:19.820776 29040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "c0f1993cae4e40f1944832fe7548ad4f"
format_stamp: "Formatted at 2026-09-28 16:34:19 on dist-test-slave-g9sh"
server_key: "2e1962ff5abd655505a49f0569f771c8"
server_key_iv: "f98632def9d4aecdecca10fa9cbbd9b6"
server_key_version: "encryptionkey@0"
I20260928 16:34:19.820881 29040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:19.844451 29040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:19.847371 29040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:19.847584 29040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:19.848202 29040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:19.849064 29040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:19.849134 29040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:19.849206 29040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:19.849255 29040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:19.858429 29040 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:37601
I20260928 16:34:19.858455 29169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:37601 every 8 connection(s)
I20260928 16:34:19.859330 29040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:34:19.860946 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 29040
I20260928 16:34:19.861052 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:34:19.861331 28139 external_mini_cluster.cc:1428] Setting key 043348d570974f7f2f8eb52f43dd5be2
Sep 28 16:34:19 dist-test-slave-g9sh krb5kdc[28665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613259, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:19.872257 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.861064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:58273 (local address 127.27.122.254:46103)
0928 16:34:19.861209 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:19.861213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:19.862171 (+ 958us) server_negotiation.cc:407] Connection header received
0928 16:34:19.863430 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:19.863435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:19.863485 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:19.863558 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:19.864642 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.865179 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.866257 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.866465 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.866543 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.868768 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:19.868771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:19.868774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:19.868798 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:19.870541 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:19.870990 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:19.870993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:19.870994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:19.871070 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:19.871350 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:19.871352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:19.871353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:19.871523 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:19.871602 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:19.872040 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 16:34:19.872134 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:34:19.873082 29172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.861556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:46103 (local address 127.27.122.195:58273)
0928 16:34:19.862039 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:19.862090 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:19.863254 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:19.863710 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:19.863716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:19.864043 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:19.864499 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:19.864513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.865284 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.865286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:19.866154 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:19.866163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.866595 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.866598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:19.866730 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.867226 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:19.867242 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:19.868616 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:19.870654 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:19.870659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:19.870670 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:19.870904 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:19.871149 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:19.871151 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:19.871152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:19.871272 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:19.871622 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:19.871626 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:19.871832 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 16:34:19.871978 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 16:34:19.872199 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":131,"threads_started":1}
I20260928 16:34:19.874306 29170 heartbeater.cc:347] Connected to a master server at 127.27.122.254:46103
I20260928 16:34:19.874527 29170 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:19.875105 29170 heartbeater.cc:511] Master 127.27.122.254:46103 requested a full tablet report, sending...
I20260928 16:34:19.876109 28710 ts_manager.cc:194] Registered new tserver with Master: c0f1993cae4e40f1944832fe7548ad4f (127.27.122.195:37601)
I20260928 16:34:19.876710 28710 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:58273
I20260928 16:34:19.886695 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:34:19.897113 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.890083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.27.122.254:46103)
0928 16:34:19.890216 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:19.890220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:19.890406 (+ 186us) server_negotiation.cc:407] Connection header received
0928 16:34:19.890497 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:19.890499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:19.890538 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:19.890604 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:19.891239 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.891732 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.892483 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.892692 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.892766 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.894001 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:19.894004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:19.894006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:19.894027 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:19.895345 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:19.895731 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:19.895734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:19.895735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:19.895807 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:19.896116 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:19.896119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:19.896121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:19.896410 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:19.896549 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:19.896788 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 16:34:19.896953 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:34:19.901387 28710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:34:19.904357 28710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:19.918569 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.912393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:45751 (local address 127.0.0.1:33932)
0928 16:34:19.913638 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:19.913653 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:19.913746 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:19.914405 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:19.914408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:19.914423 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:19.914510 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:19.914519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.915495 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.915499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:19.916809 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:19.916819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.918023 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.918027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:19.918151 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.918192 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 16:34:19.918304 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 16:34:19.918415 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":57,"thread_start_us":96,"threads_started":1}
I20260928 16:34:19.918735 29181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.912292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:36271 (local address 127.0.0.1:58458)
0928 16:34:19.912751 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:19.912794 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:19.912935 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:19.914182 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:19.914192 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:19.914236 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:19.914400 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:19.914406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.915972 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.915975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:19.917237 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:19.917249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.918296 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.918300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:19.918393 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.918415 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:34:19.918497 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 16:34:19.918575 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":36,"spinlock_wait_cycles":8448,"thread_start_us":65,"threads_started":1}
I20260928 16:34:19.918751 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.912415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58458 (local address 127.27.122.194:36271)
0928 16:34:19.913762 (+ 1347us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:19.913772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:19.913792 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:34:19.913857 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:19.913861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:19.914010 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:19.914572 (+ 562us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:19.914732 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.915825 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.917364 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.918186 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.918307 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.918428 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:19.918524 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 16:34:19.918626 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1249,"thread_start_us":133,"threads_started":1}
I20260928 16:34:19.919030 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.913086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.27.122.193:45751)
0928 16:34:19.913481 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:19.913496 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:19.913858 (+ 362us) server_negotiation.cc:407] Connection header received
0928 16:34:19.913928 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:19.913932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:19.914082 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:19.914249 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:19.914630 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.915357 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.916963 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.917898 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.918038 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.918185 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:19.918779 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 16:34:19.918895 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":82,"threads_started":1}
I20260928 16:34:19.919790 29182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.912997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:37601 (local address 127.0.0.1:45242)
0928 16:34:19.913417 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:19.913430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:19.913523 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:19.914180 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:19.914190 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:19.914228 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:19.914417 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:19.914425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.915982 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.915984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:19.917754 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:19.917764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.919373 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.919377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:19.919462 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.919482 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 16:34:19.919569 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 16:34:19.919649 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":44,"spinlock_wait_cycles":2816,"thread_start_us":116,"threads_started":1}
I20260928 16:34:19.920192 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:19.913157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.27.122.195:37601)
0928 16:34:19.913694 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:19.913706 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:19.913728 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:34:19.913807 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:19.913813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:19.914010 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:19.914178 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:19.914546 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.915781 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.917871 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:19.919241 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:19.919384 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:19.919507 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:19.919942 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 16:34:19.920042 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":157,"threads_started":1}
I20260928 16:34:19.921864 28969 tablet_service.cc:1467] Processing CreateTablet for tablet 03fe01c419ef49ebb238104f295e8423 (DEFAULT_TABLE table=test-table [id=54b7d1b05124454dba4344d60f148442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:19.921864 28835 tablet_service.cc:1467] Processing CreateTablet for tablet 03fe01c419ef49ebb238104f295e8423 (DEFAULT_TABLE table=test-table [id=54b7d1b05124454dba4344d60f148442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:19.923002 28969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fe01c419ef49ebb238104f295e8423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:19.923002 28835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fe01c419ef49ebb238104f295e8423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:19.922782 29103 tablet_service.cc:1467] Processing CreateTablet for tablet 03fe01c419ef49ebb238104f295e8423 (DEFAULT_TABLE table=test-table [id=54b7d1b05124454dba4344d60f148442]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:19.923859 29103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03fe01c419ef49ebb238104f295e8423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:19.929905 29187 tablet_bootstrap.cc:492] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f: Bootstrap starting.
I20260928 16:34:19.930584 29188 tablet_bootstrap.cc:492] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1: Bootstrap starting.
I20260928 16:34:19.930989 29189 tablet_bootstrap.cc:492] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd: Bootstrap starting.
I20260928 16:34:19.933012 29188 tablet_bootstrap.cc:654] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:19.933012 29187 tablet_bootstrap.cc:654] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:19.933270 29189 tablet_bootstrap.cc:654] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:19.934242 29189 log.cc:824] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:19.934350 29187 log.cc:824] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:19.934393 29188 log.cc:824] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:19.936091 29189 tablet_bootstrap.cc:492] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd: No bootstrap required, opened a new log
I20260928 16:34:19.936303 29189 ts_tablet_manager.cc:1397] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:34:19.936326 29187 tablet_bootstrap.cc:492] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f: No bootstrap required, opened a new log
I20260928 16:34:19.936322 29188 tablet_bootstrap.cc:492] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1: No bootstrap required, opened a new log
I20260928 16:34:19.936493 29188 ts_tablet_manager.cc:1397] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:34:19.936493 29187 ts_tablet_manager.cc:1397] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 16:34:19.939901 29189 raft_consensus.cc:348] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399f5ba0bab24b0db563ba825ed66dcd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 36271 } } peers { permanent_uuid: "eb27eb12f5c74bf591c1ff559119bca1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45751 } } peers { permanent_uuid: "c0f1993cae4e40f1944832fe7548ad4f" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37601 } }
I20260928 16:34:19.940156 29189 raft_consensus.cc:374] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:19.940232 29189 raft_consensus.cc:729] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 399f5ba0bab24b0db563ba825ed66dcd, State: Initialized, Role: FOLLOWER
I20260928 16:34:19.940696 29189 consensus_queue.cc:260] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399f5ba0bab24b0db563ba825ed66dcd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 36271 } } peers { permanent_uuid: "eb27eb12f5c74bf591c1ff559119bca1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45751 } } peers { permanent_uuid: "c0f1993cae4e40f1944832fe7548ad4f" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37601 } }
I20260928 16:34:19.940996 29188 raft_consensus.cc:348] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399f5ba0bab24b0db563ba825ed66dcd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 36271 } } peers { permanent_uuid: "eb27eb12f5c74bf591c1ff559119bca1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45751 } } peers { permanent_uuid: "c0f1993cae4e40f1944832fe7548ad4f" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37601 } }
I20260928 16:34:19.940996 29187 raft_consensus.cc:348] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399f5ba0bab24b0db563ba825ed66dcd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 36271 } } peers { permanent_uuid: "eb27eb12f5c74bf591c1ff559119bca1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45751 } } peers { permanent_uuid: "c0f1993cae4e40f1944832fe7548ad4f" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37601 } }
I20260928 16:34:19.941249 29187 raft_consensus.cc:374] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:19.941309 29188 raft_consensus.cc:374] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:19.941350 29187 raft_consensus.cc:729] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0f1993cae4e40f1944832fe7548ad4f, State: Initialized, Role: FOLLOWER
I20260928 16:34:19.941402 29188 raft_consensus.cc:729] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb27eb12f5c74bf591c1ff559119bca1, State: Initialized, Role: FOLLOWER
I20260928 16:34:19.941421 29035 heartbeater.cc:503] Master 127.27.122.254:46103 was elected leader, sending a full tablet report...
I20260928 16:34:19.941788 29189 ts_tablet_manager.cc:1428] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260928 16:34:19.941867 29188 consensus_queue.cc:260] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399f5ba0bab24b0db563ba825ed66dcd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 36271 } } peers { permanent_uuid: "eb27eb12f5c74bf591c1ff559119bca1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45751 } } peers { permanent_uuid: "c0f1993cae4e40f1944832fe7548ad4f" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37601 } }
I20260928 16:34:19.941879 29187 consensus_queue.cc:260] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399f5ba0bab24b0db563ba825ed66dcd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 36271 } } peers { permanent_uuid: "eb27eb12f5c74bf591c1ff559119bca1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45751 } } peers { permanent_uuid: "c0f1993cae4e40f1944832fe7548ad4f" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37601 } }
I20260928 16:34:19.942585 29170 heartbeater.cc:503] Master 127.27.122.254:46103 was elected leader, sending a full tablet report...
I20260928 16:34:19.942833 29188 ts_tablet_manager.cc:1428] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260928 16:34:19.942955 28902 tablet.cc:2369] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:19.943470 28901 heartbeater.cc:503] Master 127.27.122.254:46103 was elected leader, sending a full tablet report...
I20260928 16:34:19.944231 29187 ts_tablet_manager.cc:1428] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
W20260928 16:34:20.110255 29171 tablet.cc:2369] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:34:20.144145 29036 tablet.cc:2369] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:20.239015 29194 raft_consensus.cc:482] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:20.239251 29194 raft_consensus.cc:504] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399f5ba0bab24b0db563ba825ed66dcd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 36271 } } peers { permanent_uuid: "eb27eb12f5c74bf591c1ff559119bca1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45751 } } peers { permanent_uuid: "c0f1993cae4e40f1944832fe7548ad4f" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37601 } }
I20260928 16:34:20.240527 29194 leader_election.cc:288] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 399f5ba0bab24b0db563ba825ed66dcd (127.27.122.194:36271), eb27eb12f5c74bf591c1ff559119bca1 (127.27.122.193:45751)
I20260928 16:34:20.241722 29193 raft_consensus.cc:482] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:20.241921 29193 raft_consensus.cc:504] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399f5ba0bab24b0db563ba825ed66dcd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 36271 } } peers { permanent_uuid: "eb27eb12f5c74bf591c1ff559119bca1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45751 } } peers { permanent_uuid: "c0f1993cae4e40f1944832fe7548ad4f" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37601 } }
I20260928 16:34:20.243312 29193 leader_election.cc:288] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb27eb12f5c74bf591c1ff559119bca1 (127.27.122.193:45751), c0f1993cae4e40f1944832fe7548ad4f (127.27.122.195:37601)
I20260928 16:34:20.245857 29172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:20.240754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:36271 (local address 127.27.122.195:58333)
0928 16:34:20.240891 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:20.240926 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:20.241053 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:20.241755 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:20.241757 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:20.241772 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:20.241955 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:20.241962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.242897 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.242900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:20.244297 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:20.244309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.245425 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.245430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:20.245532 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:20.245555 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:34:20.245651 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 16:34:20.245720 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":148}
I20260928 16:34:20.245903 29185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:20.240897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:57151 (local address 127.27.122.193:45751)
0928 16:34:20.241074 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:20.241081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:20.241413 (+ 332us) server_negotiation.cc:407] Connection header received
0928 16:34:20.241499 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:20.241502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:20.241554 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:20.241638 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:20.242033 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.242599 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.244540 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.245199 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.245291 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:20.245350 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:20.245708 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 16:34:20.245781 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:34:20.246325 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:20.240810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:45751 (local address 127.27.122.195:57151)
0928 16:34:20.241299 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:20.241312 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:20.241422 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:20.241673 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:20.241676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:20.241695 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:20.241852 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:20.241861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.242727 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.242732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:20.244418 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:20.244426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.245438 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.245441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:20.245572 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:20.245594 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:34:20.246093 (+ 499us) client_negotiation.cc:240] Negotiation successful
0928 16:34:20.246178 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":65,"thread_start_us":172,"threads_started":1}
I20260928 16:34:20.246553 29183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:20.240998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:58333 (local address 127.27.122.194:36271)
0928 16:34:20.241162 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:20.241166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:20.241182 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:34:20.241244 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:20.241247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:20.241302 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:20.241407 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:20.242061 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.242779 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.244406 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.245183 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.245310 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:20.245356 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:20.246359 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0928 16:34:20.246442 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":71}
I20260928 16:34:20.247077 28989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fe01c419ef49ebb238104f295e8423" candidate_uuid: "c0f1993cae4e40f1944832fe7548ad4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "399f5ba0bab24b0db563ba825ed66dcd" is_pre_election: true
I20260928 16:34:20.247107 28855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fe01c419ef49ebb238104f295e8423" candidate_uuid: "c0f1993cae4e40f1944832fe7548ad4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb27eb12f5c74bf591c1ff559119bca1" is_pre_election: true
I20260928 16:34:20.247290 28989 raft_consensus.cc:2454] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f1993cae4e40f1944832fe7548ad4f in term 0.
I20260928 16:34:20.247391 28855 raft_consensus.cc:2454] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0f1993cae4e40f1944832fe7548ad4f in term 0.
I20260928 16:34:20.247879 29057 leader_election.cc:302] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0f1993cae4e40f1944832fe7548ad4f, eb27eb12f5c74bf591c1ff559119bca1; no voters:
I20260928 16:34:20.248153 29194 raft_consensus.cc:2781] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:20.248306 29194 raft_consensus.cc:482] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:20.248373 29194 raft_consensus.cc:3037] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:20.248494 29197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:20.243435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:45751 (local address 127.27.122.194:44515)
0928 16:34:20.243721 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:20.243736 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:20.243828 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:20.244376 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:20.244379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:20.244401 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:20.244681 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:20.244687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.246006 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.246010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:20.247178 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:20.247188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.248075 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.248079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:20.248180 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:20.248203 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:34:20.248270 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 16:34:20.248342 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":112,"thread_start_us":81,"threads_started":1}
I20260928 16:34:20.248834 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:20.243648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:44515 (local address 127.27.122.193:45751)
0928 16:34:20.244100 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:20.244104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:20.244118 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:34:20.244161 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:20.244165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:20.244216 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:20.244461 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:20.244821 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.245858 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.247319 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.247950 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.248080 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:20.248121 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:20.248638 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 16:34:20.248706 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":61,"threads_started":1}
I20260928 16:34:20.249398 28855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fe01c419ef49ebb238104f295e8423" candidate_uuid: "399f5ba0bab24b0db563ba825ed66dcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb27eb12f5c74bf591c1ff559119bca1" is_pre_election: true
I20260928 16:34:20.249531 28855 raft_consensus.cc:2454] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 399f5ba0bab24b0db563ba825ed66dcd in term 0.
I20260928 16:34:20.249547 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:20.244291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:44617 (local address 127.27.122.195:37601)
0928 16:34:20.244410 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:20.244416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:20.244630 (+ 214us) server_negotiation.cc:407] Connection header received
0928 16:34:20.244797 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:20.244800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:20.244836 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:20.244876 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:20.245282 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.245953 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.248330 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.248880 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.249003 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:20.249058 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:20.249372 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 16:34:20.249440 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 16:34:20.249572 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:20.243981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:37601 (local address 127.27.122.194:44617)
0928 16:34:20.244540 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:20.244555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:20.244693 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:20.245030 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:20.245033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:20.245049 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:20.245157 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:20.245163 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.246098 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.246102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:20.248192 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:20.248201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.249160 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.249164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:20.249246 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:20.249261 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 16:34:20.249355 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 16:34:20.249417 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260928 16:34:20.249957 28923 leader_election.cc:302] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 399f5ba0bab24b0db563ba825ed66dcd, eb27eb12f5c74bf591c1ff559119bca1; no voters:
I20260928 16:34:20.249998 29194 raft_consensus.cc:504] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399f5ba0bab24b0db563ba825ed66dcd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 36271 } } peers { permanent_uuid: "eb27eb12f5c74bf591c1ff559119bca1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45751 } } peers { permanent_uuid: "c0f1993cae4e40f1944832fe7548ad4f" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37601 } }
I20260928 16:34:20.250216 29193 raft_consensus.cc:2781] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:20.250329 29193 raft_consensus.cc:482] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:20.250336 29194 leader_election.cc:288] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [CANDIDATE]: Term 1 election: Requested vote from peers 399f5ba0bab24b0db563ba825ed66dcd (127.27.122.194:36271), eb27eb12f5c74bf591c1ff559119bca1 (127.27.122.193:45751)
I20260928 16:34:20.250427 29193 raft_consensus.cc:3037] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:20.250602 29124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fe01c419ef49ebb238104f295e8423" candidate_uuid: "399f5ba0bab24b0db563ba825ed66dcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f1993cae4e40f1944832fe7548ad4f" is_pre_election: true
I20260928 16:34:20.250636 28989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fe01c419ef49ebb238104f295e8423" candidate_uuid: "c0f1993cae4e40f1944832fe7548ad4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "399f5ba0bab24b0db563ba825ed66dcd"
I20260928 16:34:20.250824 29124 raft_consensus.cc:2379] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 399f5ba0bab24b0db563ba825ed66dcd in current term 1: Already voted for candidate c0f1993cae4e40f1944832fe7548ad4f in this term.
I20260928 16:34:20.251477 28855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fe01c419ef49ebb238104f295e8423" candidate_uuid: "c0f1993cae4e40f1944832fe7548ad4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb27eb12f5c74bf591c1ff559119bca1"
I20260928 16:34:20.251605 28855 raft_consensus.cc:3037] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:20.251614 29193 raft_consensus.cc:504] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399f5ba0bab24b0db563ba825ed66dcd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 36271 } } peers { permanent_uuid: "eb27eb12f5c74bf591c1ff559119bca1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45751 } } peers { permanent_uuid: "c0f1993cae4e40f1944832fe7548ad4f" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37601 } }
I20260928 16:34:20.251847 28989 raft_consensus.cc:2379] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c0f1993cae4e40f1944832fe7548ad4f in current term 1: Already voted for candidate 399f5ba0bab24b0db563ba825ed66dcd in this term.
I20260928 16:34:20.251909 29193 leader_election.cc:288] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [CANDIDATE]: Term 1 election: Requested vote from peers eb27eb12f5c74bf591c1ff559119bca1 (127.27.122.193:45751), c0f1993cae4e40f1944832fe7548ad4f (127.27.122.195:37601)
I20260928 16:34:20.252210 29124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fe01c419ef49ebb238104f295e8423" candidate_uuid: "399f5ba0bab24b0db563ba825ed66dcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0f1993cae4e40f1944832fe7548ad4f"
I20260928 16:34:20.252306 29124 raft_consensus.cc:2379] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 399f5ba0bab24b0db563ba825ed66dcd in current term 1: Already voted for candidate c0f1993cae4e40f1944832fe7548ad4f in this term.
I20260928 16:34:20.252846 28854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03fe01c419ef49ebb238104f295e8423" candidate_uuid: "399f5ba0bab24b0db563ba825ed66dcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb27eb12f5c74bf591c1ff559119bca1"
I20260928 16:34:20.252995 28855 raft_consensus.cc:2454] T 03fe01c419ef49ebb238104f295e8423 P eb27eb12f5c74bf591c1ff559119bca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0f1993cae4e40f1944832fe7548ad4f in term 1.
I20260928 16:34:20.253247 29057 leader_election.cc:302] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0f1993cae4e40f1944832fe7548ad4f, eb27eb12f5c74bf591c1ff559119bca1; no voters: 399f5ba0bab24b0db563ba825ed66dcd
I20260928 16:34:20.253391 28923 leader_election.cc:302] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 399f5ba0bab24b0db563ba825ed66dcd; no voters: c0f1993cae4e40f1944832fe7548ad4f, eb27eb12f5c74bf591c1ff559119bca1
I20260928 16:34:20.253448 29194 raft_consensus.cc:2781] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:20.253609 29193 raft_consensus.cc:2726] T 03fe01c419ef49ebb238104f295e8423 P 399f5ba0bab24b0db563ba825ed66dcd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 16:34:20.253780 29194 raft_consensus.cc:686] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [term 1 LEADER]: Becoming Leader. State: Replica: c0f1993cae4e40f1944832fe7548ad4f, State: Running, Role: LEADER
I20260928 16:34:20.254266 29194 consensus_queue.cc:237] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399f5ba0bab24b0db563ba825ed66dcd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 36271 } } peers { permanent_uuid: "eb27eb12f5c74bf591c1ff559119bca1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45751 } } peers { permanent_uuid: "c0f1993cae4e40f1944832fe7548ad4f" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37601 } }
I20260928 16:34:20.257370 28710 catalog_manager.cc:5495] T 03fe01c419ef49ebb238104f295e8423 P c0f1993cae4e40f1944832fe7548ad4f reported cstate change: term changed from 0 to 1, leader changed from <none> to c0f1993cae4e40f1944832fe7548ad4f (127.27.122.195). New cstate: current_term: 1 leader_uuid: "c0f1993cae4e40f1944832fe7548ad4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "399f5ba0bab24b0db563ba825ed66dcd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb27eb12f5c74bf591c1ff559119bca1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0f1993cae4e40f1944832fe7548ad4f" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37601 } health_report { overall_health: HEALTHY } } }
Sep 28 16:34:20 dist-test-slave-g9sh krb5kdc[28665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.193@KRBTEST.COM
I20260928 16:34:20.285961 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:20.277396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33938 (local address 127.27.122.193:45751)
0928 16:34:20.277545 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:20.277549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:20.277721 (+ 172us) server_negotiation.cc:407] Connection header received
0928 16:34:20.277826 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:20.277828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:20.277870 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:20.277938 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:20.278602 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.279099 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.279916 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.280125 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.280198 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:20.282596 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:20.282607 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:20.282622 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:20.282644 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:20.284365 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:20.284752 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:20.284757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:20.284759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:20.284841 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:20.285117 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:20.285119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:20.285121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:20.285349 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:20.285499 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:20.285657 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 16:34:20.285745 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 16:34:20 dist-test-slave-g9sh krb5kdc[28665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613260, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:34:20 dist-test-slave-g9sh krb5kdc[28665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613260, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.122.193@KRBTEST.COM
I20260928 16:34:20.311904 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:20.303886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33946 (local address 127.27.122.193:45751)
0928 16:34:20.304061 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:20.304064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:20.304131 (+ 67us) server_negotiation.cc:407] Connection header received
0928 16:34:20.304234 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:20.304237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:20.304275 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:20.304344 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:20.305045 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.305561 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.306310 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.306537 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.306608 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:20.309017 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:20.309020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:20.309022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:20.309045 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:20.310356 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:20.310755 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:20.310757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:20.310759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:20.310838 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:20.311072 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:20.311074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:20.311075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:20.311230 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:20.311333 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:20.311594 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 16:34:20.311722 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 16:34:20 dist-test-slave-g9sh krb5kdc[28665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:34:20 dist-test-slave-g9sh krb5kdc[28665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.193@KRBTEST.COM
I20260928 16:34:20.337098 29198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:20.329105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33950 (local address 127.27.122.193:45751)
0928 16:34:20.329249 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:20.329253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:20.329398 (+ 145us) server_negotiation.cc:407] Connection header received
0928 16:34:20.329503 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:20.329505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:20.329546 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:20.329615 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:20.330220 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.330720 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.331526 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:20.331764 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:20.331857 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:20.334206 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:20.334209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:20.334211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:20.334235 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:20.335556 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:20.335930 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:20.335933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:20.335935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:20.336014 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:20.336248 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:20.336250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:20.336252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:20.336413 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:20.336544 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:20.336743 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 16:34:20.336860 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 16:34:20.338523 28815 tablet_service.cc:2908] Rejecting scan request for tablet 03fe01c419ef49ebb238104f295e8423: Uninitialized: clean time has not yet been initialized
I20260928 16:34:20.340477 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 28772
I20260928 16:34:20.347296 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 28906
I20260928 16:34:20.353025 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 29040
I20260928 16:34:20.358666 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 28678
2026-09-28T16:34:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2516 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:20 dist-test-slave-g9sh krb5kdc[29214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:20 dist-test-slave-g9sh krb5kdc[29214](info): set up 2 sockets
Sep 28 16:34:20 dist-test-slave-g9sh krb5kdc[29214](info): commencing operation
krb5kdc: starting...
W20260928 16:34:21.406948 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:21 dist-test-slave-g9sh krb5kdc[29214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:34:21.524149 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:34995
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:34995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:21.631954 29227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:21.632196 29227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:21.632236 29227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:21.636240 29227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:21.636291 29227 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:34:21.636307 29227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:21.636324 29227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:21.636369 29227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:21.636407 29227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:21.641523 29227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:34995
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:34995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:21.642490 29227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:21.643487 29227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:21.651780 29233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:21.651835 29235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:21.651796 29232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:21.652112 29227 server_base.cc:1034] running on GCE node
I20260928 16:34:21.652715 29227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:21.653796 29227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:21.654983 29227 hybrid_clock.cc:648] HybridClock initialized: now 1790613261654960 us; error 34 us; skew 500 ppm
Sep 28 16:34:21 dist-test-slave-g9sh krb5kdc[29214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613261, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:21.662742 29227 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:34:21.663552 29227 webserver.cc:460] Webserver started at http://127.27.122.254:34639/ using document root <none> and password file <none>
I20260928 16:34:21.664072 29227 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:21.664127 29227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:21.664287 29227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:21.666307 29227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "384ca5ebebbd4f8088298f06813c43db"
format_stamp: "Formatted at 2026-09-28 16:34:21 on dist-test-slave-g9sh"
server_key: "88ba9937eb498e295657660026c6bb68"
server_key_iv: "11cefb6dbf62b00bae89ef5b53012dd4"
server_key_version: "encryptionkey@0"
I20260928 16:34:21.666828 29227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "384ca5ebebbd4f8088298f06813c43db"
format_stamp: "Formatted at 2026-09-28 16:34:21 on dist-test-slave-g9sh"
server_key: "88ba9937eb498e295657660026c6bb68"
server_key_iv: "11cefb6dbf62b00bae89ef5b53012dd4"
server_key_version: "encryptionkey@0"
I20260928 16:34:21.670689 29227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:34:21.673156 29242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:21.674484 29227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:21.674584 29227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "384ca5ebebbd4f8088298f06813c43db"
format_stamp: "Formatted at 2026-09-28 16:34:21 on dist-test-slave-g9sh"
server_key: "88ba9937eb498e295657660026c6bb68"
server_key_iv: "11cefb6dbf62b00bae89ef5b53012dd4"
server_key_version: "encryptionkey@0"
I20260928 16:34:21.674657 29227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:21.698755 29227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:21.701498 29227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:21.701646 29227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:21.709276 29227 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:34995
I20260928 16:34:21.709287 29304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:34995 every 8 connection(s)
I20260928 16:34:21.710245 29227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:21.713011 29305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:21.718401 29305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db: Bootstrap starting.
I20260928 16:34:21.720283 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 29227
I20260928 16:34:21.720377 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:21.720629 28139 external_mini_cluster.cc:1428] Setting key a290b31dc163a4037c7d4c2a0cec9142
I20260928 16:34:21.721110 29305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:21.721992 29305 log.cc:824] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:21.724175 29305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db: No bootstrap required, opened a new log
I20260928 16:34:21.728488 29305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384ca5ebebbd4f8088298f06813c43db" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34995 } }
I20260928 16:34:21.728752 29305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:21.728819 29305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384ca5ebebbd4f8088298f06813c43db, State: Initialized, Role: FOLLOWER
I20260928 16:34:21.729321 29305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384ca5ebebbd4f8088298f06813c43db" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34995 } }
I20260928 16:34:21.729445 29305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:21.729527 29305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:21.729655 29305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [term 0 FOLLOWER]: Advancing to term 1
Sep 28 16:34:21 dist-test-slave-g9sh krb5kdc[29214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:21.730647 29305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384ca5ebebbd4f8088298f06813c43db" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34995 } }
I20260928 16:34:21.731020 29305 leader_election.cc:302] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 384ca5ebebbd4f8088298f06813c43db; no voters:
I20260928 16:34:21.731398 29305 leader_election.cc:288] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:21.731559 29310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:21.731803 29310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [term 1 LEADER]: Becoming Leader. State: Replica: 384ca5ebebbd4f8088298f06813c43db, State: Running, Role: LEADER
I20260928 16:34:21.732132 29310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384ca5ebebbd4f8088298f06813c43db" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34995 } }
I20260928 16:34:21.732785 29305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:21.733268 29310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 384ca5ebebbd4f8088298f06813c43db. Latest consensus state: current_term: 1 leader_uuid: "384ca5ebebbd4f8088298f06813c43db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384ca5ebebbd4f8088298f06813c43db" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34995 } } }
I20260928 16:34:21.733392 29310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:21.735340 29312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "384ca5ebebbd4f8088298f06813c43db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "384ca5ebebbd4f8088298f06813c43db" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34995 } } }
I20260928 16:34:21.735445 29312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:21.735540 29314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:21.736706 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:21.721798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.27.122.254:34995)
0928 16:34:21.722268 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:21.722291 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:21.722343 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:34:21.723139 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:21.723162 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:21.723556 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:21.723929 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:21.724649 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:21.726242 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:21.727148 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:21.727875 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:21.728040 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:21.730750 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:21.730757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:21.730773 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:21.730803 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:21.732969 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:21.733590 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:21.733596 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:21.733604 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:21.733768 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:21.734125 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:21.734130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:21.734132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:21.734684 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:21.734884 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:21.735784 (+ 900us) server_negotiation.cc:299] Negotiation successful
0928 16:34:21.736012 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":143,"threads_started":1}
I20260928 16:34:21.738495 29314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:21.743947 29314 catalog_manager.cc:1347] Generated new cluster ID: 78980dd8547446fba16e1f6fadf30324
I20260928 16:34:21.744016 29314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:21.752345 29314 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:21.753338 29314 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:21.759790 29314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db: Generated new TSK 0
I20260928 16:34:21.760452 29314 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:34:21.791640 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34995
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:21.917273 29331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:21.917503 29331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:21.917572 29331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:21.921522 29331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:21.921571 29331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:21.921607 29331 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:34:21.921677 29331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:21.926736 29331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.122.254:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:21.927772 29331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:21.928795 29331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:21.938400 29336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:21.938413 29339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:21.938413 29337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:21.938464 29331 server_base.cc:1034] running on GCE node
I20260928 16:34:21.939040 29331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:21.939656 29331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:21.940838 29331 hybrid_clock.cc:648] HybridClock initialized: now 1790613261940800 us; error 50 us; skew 500 ppm
Sep 28 16:34:21 dist-test-slave-g9sh krb5kdc[29214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613261, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:21.943476 29331 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:34:21.944274 29331 webserver.cc:460] Webserver started at http://127.27.122.193:46039/ using document root <none> and password file <none>
I20260928 16:34:21.944829 29331 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:21.944938 29331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:21.945176 29331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:21.947124 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "c486231190c4436ca2a4ec5e5e1ef2e9"
format_stamp: "Formatted at 2026-09-28 16:34:21 on dist-test-slave-g9sh"
server_key: "553ae4c094c2e8d1a3ec61d809a82efb"
server_key_iv: "978a5ede6e498382505bb55cc5eea789"
server_key_version: "encryptionkey@0"
I20260928 16:34:21.947685 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "c486231190c4436ca2a4ec5e5e1ef2e9"
format_stamp: "Formatted at 2026-09-28 16:34:21 on dist-test-slave-g9sh"
server_key: "553ae4c094c2e8d1a3ec61d809a82efb"
server_key_iv: "978a5ede6e498382505bb55cc5eea789"
server_key_version: "encryptionkey@0"
I20260928 16:34:21.951347 29331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 16:34:21.953739 29346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:21.954972 29331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:21.955106 29331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "c486231190c4436ca2a4ec5e5e1ef2e9"
format_stamp: "Formatted at 2026-09-28 16:34:21 on dist-test-slave-g9sh"
server_key: "553ae4c094c2e8d1a3ec61d809a82efb"
server_key_iv: "978a5ede6e498382505bb55cc5eea789"
server_key_version: "encryptionkey@0"
I20260928 16:34:21.955214 29331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:21.976287 29331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:21.979496 29331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:21.979698 29331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:21.981148 29331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:21.981220 29331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:21.981290 29331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:21.981334 29331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:34:21 dist-test-slave-g9sh krb5kdc[29214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613261, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:21.992317 29331 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:33059
I20260928 16:34:21.992489 29461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:33059 every 8 connection(s)
I20260928 16:34:21.993279 29331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:21.996161 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:21.982019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:43533 (local address 127.27.122.254:34995)
0928 16:34:21.982146 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:21.982150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:21.983204 (+ 1054us) server_negotiation.cc:407] Connection header received
0928 16:34:21.984170 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:21.984173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:21.984227 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:21.984284 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:21.986282 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:21.987278 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:21.989233 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:21.989532 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:21.989630 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:21.992609 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:21.992613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:21.992616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:21.992669 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:21.994391 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:21.994857 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:21.994859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:21.994860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:21.994938 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:21.995240 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:21.995243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:21.995244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:21.995418 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:21.995496 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:21.995929 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 16:34:21.996041 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:34:21.997048 29355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:21.982560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:34995 (local address 127.27.122.193:43533)
0928 16:34:21.983050 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:21.983092 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:21.983913 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:21.984767 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:21.984777 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:21.985284 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:21.985999 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:21.986025 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:21.987436 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:21.987441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:21.989088 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:21.989102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:21.989714 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:21.989721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:21.989940 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:21.990607 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:21.990631 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:21.992392 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:21.994504 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:21.994508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:21.994519 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:21.994768 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:21.995027 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:21.995029 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:21.995031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:21.995160 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:21.995504 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:21.995509 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:21.995732 (+ 223us) client_negotiation.cc:769] Sending connection context
0928 16:34:21.995882 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 16:34:21.996123 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":139,"threads_started":1}
I20260928 16:34:21.998183 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 29331
I20260928 16:34:21.998311 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:34:21.998433 29462 heartbeater.cc:347] Connected to a master server at 127.27.122.254:34995
I20260928 16:34:21.998585 29462 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:21.998653 28139 external_mini_cluster.cc:1428] Setting key 7f10ceeabee8c2fb89c64bf2238204d1
I20260928 16:34:21.999163 29462 heartbeater.cc:511] Master 127.27.122.254:34995 requested a full tablet report, sending...
I20260928 16:34:22.000775 29258 ts_manager.cc:194] Registered new tserver with Master: c486231190c4436ca2a4ec5e5e1ef2e9 (127.27.122.193:33059)
I20260928 16:34:22.001987 29258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:43533
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
I20260928 16:34:22.011162 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.000670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:57771 (local address 127.27.122.254:34995)
0928 16:34:22.002548 (+ 1878us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.002581 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.002598 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:34:22.002633 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.002635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.002675 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.002761 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:22.004608 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.005515 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.006366 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.006589 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.006701 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.007788 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:22.007791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:22.007794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:22.007825 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:22.009415 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.009843 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.009847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.009849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.009980 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.010297 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.010301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.010302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.010546 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:22.010657 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.010944 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.011049 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1822}
I20260928 16:34:22.011334 29468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.000683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:34995 (local address 127.27.122.193:57771)
0928 16:34:22.001007 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:22.001022 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:22.001111 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:22.002808 (+ 1697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:22.004112 (+ 1304us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:22.004359 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:22.004463 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.004469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.005659 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.005662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.006238 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.006246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.006704 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.006708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.006819 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.007321 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:22.007332 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:22.007642 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:22.009538 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:22.009541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:22.009543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:22.009740 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:22.010107 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:22.010110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:22.010112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:22.010201 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:22.010658 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:22.010662 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:22.010759 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 16:34:22.010872 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 16:34:22.011024 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:34:22.025193 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34995
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:22.133626 29471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:22.133919 29471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:22.133991 29471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:22.138108 29471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:22.138201 29471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:22.138274 29471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:34:22.138360 29471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:22.143455 29471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.122.254:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:22.144438 29471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:22.145521 29471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:22.154989 29479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:22.154999 29477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:22.155041 29476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:22.155074 29471 server_base.cc:1034] running on GCE node
I20260928 16:34:22.155618 29471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:22.156239 29471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:22.157419 29471 hybrid_clock.cc:648] HybridClock initialized: now 1790613262157385 us; error 50 us; skew 500 ppm
Sep 28 16:34:22 dist-test-slave-g9sh krb5kdc[29214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613262, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:22.160225 29471 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:34:22.161074 29471 webserver.cc:460] Webserver started at http://127.27.122.194:37049/ using document root <none> and password file <none>
I20260928 16:34:22.161628 29471 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:22.161687 29471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:22.161947 29471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:22.163880 29471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "ccad18f5ab7e4f19a4dc53815bf4c088"
format_stamp: "Formatted at 2026-09-28 16:34:22 on dist-test-slave-g9sh"
server_key: "bb82c397a5c31467a70374e815e33c1b"
server_key_iv: "1a6ff2c8ee5f1b79cb064eea86262062"
server_key_version: "encryptionkey@0"
I20260928 16:34:22.164419 29471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "ccad18f5ab7e4f19a4dc53815bf4c088"
format_stamp: "Formatted at 2026-09-28 16:34:22 on dist-test-slave-g9sh"
server_key: "bb82c397a5c31467a70374e815e33c1b"
server_key_iv: "1a6ff2c8ee5f1b79cb064eea86262062"
server_key_version: "encryptionkey@0"
I20260928 16:34:22.168000 29471 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:34:22.170318 29486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:22.171500 29471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:34:22.171635 29471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "ccad18f5ab7e4f19a4dc53815bf4c088"
format_stamp: "Formatted at 2026-09-28 16:34:22 on dist-test-slave-g9sh"
server_key: "bb82c397a5c31467a70374e815e33c1b"
server_key_iv: "1a6ff2c8ee5f1b79cb064eea86262062"
server_key_version: "encryptionkey@0"
I20260928 16:34:22.171741 29471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:22.183151 29471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:22.185972 29471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:22.186168 29471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:22.187546 29471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:22.187628 29471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:22.187700 29471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:22.187752 29471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:34:22 dist-test-slave-g9sh krb5kdc[29214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613262, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:22.198606 29471 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:35721
I20260928 16:34:22.199489 29471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:22.200263 29601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:35721 every 8 connection(s)
I20260928 16:34:22.200598 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 29471
I20260928 16:34:22.200734 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:34:22.201054 28139 external_mini_cluster.cc:1428] Setting key 91a8e9bd8fe93e4d8d295ec23fc91631
I20260928 16:34:22.201493 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.188410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:48429 (local address 127.27.122.254:34995)
0928 16:34:22.188562 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.188567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.189516 (+ 949us) server_negotiation.cc:407] Connection header received
0928 16:34:22.190814 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.190819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.190886 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.191009 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:22.192603 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.193466 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.194697 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.194943 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.195053 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.197492 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:22.197495 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:22.197497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:22.197520 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:22.199121 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.199771 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.199776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.199778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.199860 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.200298 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.200301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.200302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.200532 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:22.200645 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.201201 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.201323 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:34:22.202383 29495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.188794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:34995 (local address 127.27.122.194:48429)
0928 16:34:22.189339 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:22.189396 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:22.190553 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:22.191236 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:22.191248 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:22.191705 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:22.192423 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.192448 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.193588 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.193592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.194564 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.194579 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.195129 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.195136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.195292 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.195863 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:22.195880 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:22.197321 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:22.199240 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:22.199246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:22.199260 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:22.199661 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:22.199989 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:22.199992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:22.199994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:22.200179 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:22.200645 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:22.200651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:22.200954 (+ 303us) client_negotiation.cc:769] Sending connection context
0928 16:34:22.201157 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 16:34:22.201453 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":157,"threads_started":1}
I20260928 16:34:22.203650 29602 heartbeater.cc:347] Connected to a master server at 127.27.122.254:34995
I20260928 16:34:22.203792 29602 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:22.204349 29602 heartbeater.cc:511] Master 127.27.122.254:34995 requested a full tablet report, sending...
I20260928 16:34:22.205471 29258 ts_manager.cc:194] Registered new tserver with Master: ccad18f5ab7e4f19a4dc53815bf4c088 (127.27.122.194:35721)
I20260928 16:34:22.206179 29258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:48429
I20260928 16:34:22.213548 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.205923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:34995 (local address 127.27.122.194:55667)
0928 16:34:22.206212 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:22.206227 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:22.206356 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:22.206662 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:22.206665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:22.206857 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:22.206954 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.206960 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.207926 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.207928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.208468 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.208476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.209125 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.209127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.209195 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.209756 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:22.209767 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:22.210059 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:22.212024 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:22.212027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:22.212028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:22.212212 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:22.212547 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:22.212549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:22.212551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:22.212628 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:22.213164 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:22.213166 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:22.213238 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 16:34:22.213319 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 16:34:22.213420 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":107,"threads_started":1}
I20260928 16:34:22.213716 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.205898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:55667 (local address 127.27.122.254:34995)
0928 16:34:22.206015 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.206022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.206294 (+ 272us) server_negotiation.cc:407] Connection header received
0928 16:34:22.206466 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.206472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.206532 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.206626 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:22.207069 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.207798 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.208593 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.209014 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.209139 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.210205 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:22.210212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:22.210215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:22.210249 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:22.211904 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.212333 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.212338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.212339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.212438 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.212719 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.212726 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.212729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.213067 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:22.213187 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.213412 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.213538 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:34:22.232265 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34995
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:22.342051 29611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:22.342314 29611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:22.342386 29611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:22.346342 29611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:22.346426 29611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:22.346494 29611 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:34:22.346585 29611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:22.351660 29611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.122.254:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:22.352630 29611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:22.353623 29611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:22.363160 29619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:22.363235 29611 server_base.cc:1034] running on GCE node
W20260928 16:34:22.363142 29617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:22.363163 29616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:22.363832 29611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:22.364487 29611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:22.365669 29611 hybrid_clock.cc:648] HybridClock initialized: now 1790613262365657 us; error 24 us; skew 500 ppm
Sep 28 16:34:22 dist-test-slave-g9sh krb5kdc[29214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613262, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:22.368342 29611 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:34:22.369230 29611 webserver.cc:460] Webserver started at http://127.27.122.195:36719/ using document root <none> and password file <none>
I20260928 16:34:22.369732 29611 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:22.369791 29611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:22.369947 29611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:22.371832 29611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "d392313b38d6412092161eac7a4a16bf"
format_stamp: "Formatted at 2026-09-28 16:34:22 on dist-test-slave-g9sh"
server_key: "0a35238ab791ba65d1853f4ee0ef4703"
server_key_iv: "eae8efde1eccab9a3684385c8385eeee"
server_key_version: "encryptionkey@0"
I20260928 16:34:22.372323 29611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "d392313b38d6412092161eac7a4a16bf"
format_stamp: "Formatted at 2026-09-28 16:34:22 on dist-test-slave-g9sh"
server_key: "0a35238ab791ba65d1853f4ee0ef4703"
server_key_iv: "eae8efde1eccab9a3684385c8385eeee"
server_key_version: "encryptionkey@0"
I20260928 16:34:22.375990 29611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:34:22.378485 29626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:22.379786 29611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:34:22.379885 29611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "d392313b38d6412092161eac7a4a16bf"
format_stamp: "Formatted at 2026-09-28 16:34:22 on dist-test-slave-g9sh"
server_key: "0a35238ab791ba65d1853f4ee0ef4703"
server_key_iv: "eae8efde1eccab9a3684385c8385eeee"
server_key_version: "encryptionkey@0"
I20260928 16:34:22.379964 29611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:22.400138 29611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:22.403193 29611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:22.403335 29611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:22.405153 29611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:22.405205 29611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:22.405244 29611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:22.405299 29611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:34:22 dist-test-slave-g9sh krb5kdc[29214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613262, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:22.415776 29611 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:34009
I20260928 16:34:22.415835 29741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:34009 every 8 connection(s)
I20260928 16:34:22.416795 29611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:34:22.418277 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 29611
I20260928 16:34:22.418397 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:34:22.418771 28139 external_mini_cluster.cc:1428] Setting key 201f09a09dbb904ffbaf1564cac56d29
I20260928 16:34:22.419824 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.405936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:35379 (local address 127.27.122.254:34995)
0928 16:34:22.406108 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.406114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.406885 (+ 771us) server_negotiation.cc:407] Connection header received
0928 16:34:22.407888 (+ 1003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.407892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.407981 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.408049 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:22.409805 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.410701 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.412480 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.412803 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.412991 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.416035 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:22.416039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:22.416041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:22.416068 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:22.417605 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.418095 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.418098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.418101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.418189 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.418608 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.418612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.418613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.418842 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:22.418960 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.419511 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.419632 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:34:22.420543 29635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.406295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:34995 (local address 127.27.122.195:35379)
0928 16:34:22.406726 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:22.406773 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:22.407647 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:22.408473 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:22.408484 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:22.408953 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:22.409611 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.409635 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.410845 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.410850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.412351 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.412362 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.413032 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.413039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.413181 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.413824 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:22.413850 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:22.415848 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:22.417723 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:22.417729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:22.417742 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:22.417991 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:22.418279 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:22.418282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:22.418284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:22.418458 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:22.418948 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:22.418955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:22.419247 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 16:34:22.419468 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 16:34:22.419773 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":126,"threads_started":1}
I20260928 16:34:22.421700 29742 heartbeater.cc:347] Connected to a master server at 127.27.122.254:34995
I20260928 16:34:22.421850 29742 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:22.422314 29742 heartbeater.cc:511] Master 127.27.122.254:34995 requested a full tablet report, sending...
I20260928 16:34:22.423274 29259 ts_manager.cc:194] Registered new tserver with Master: d392313b38d6412092161eac7a4a16bf (127.27.122.195:34009)
I20260928 16:34:22.424041 29259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:35379
I20260928 16:34:22.430342 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.423834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:48183 (local address 127.27.122.254:34995)
0928 16:34:22.423993 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.423996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.424070 (+ 74us) server_negotiation.cc:407] Connection header received
0928 16:34:22.424235 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.424237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.424274 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.424365 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:22.424874 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.425393 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.426199 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.426431 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.426547 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.427610 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:22.427614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:22.427616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:22.427638 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:22.428763 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.429210 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.429213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.429215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.429291 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.429566 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.429570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.429572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.429743 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:22.429850 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.430099 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.430201 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 16:34:22.430359 29748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.423717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:34995 (local address 127.27.122.195:48183)
0928 16:34:22.424003 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:22.424018 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:22.424114 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:22.424471 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:22.424474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:22.424671 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:22.424760 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.424765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.425524 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.425528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.426083 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.426091 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.426560 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.426564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.426661 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.427143 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:22.427154 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:22.427431 (+ 277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:22.428878 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:22.428882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:22.428884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:22.429104 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:22.429386 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:22.429390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:22.429391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:22.429475 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:22.429842 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:22.429845 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:22.429939 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 16:34:22.430042 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 16:34:22.430188 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
I20260928 16:34:22.432467 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:34:22 dist-test-slave-g9sh krb5kdc[29214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613262, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:34:22 dist-test-slave-g9sh krb5kdc[29214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613262, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:22.457690 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.449810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.27.122.254:34995)
0928 16:34:22.449932 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.449936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.449949 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:34:22.449985 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.449987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.450026 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.450096 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:22.450757 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.451264 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.452098 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.452301 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.452377 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.454796 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:22.454799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:22.454801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:22.454826 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:22.456129 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.456547 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.456550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.456551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.456632 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.456873 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.456875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.456876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.457139 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:22.457246 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.457465 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.457559 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:34:22.460706 29259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:34:22.463786 29259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:22.478413 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.473090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.27.122.194:35721)
0928 16:34:22.473917 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.473937 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.473956 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:34:22.474019 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.474023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.474157 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.474309 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:22.474531 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.475509 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.477009 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.477750 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.477908 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.478047 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.478173 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.478259 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":115,"threads_started":1}
I20260928 16:34:22.478444 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.472888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:35721 (local address 127.0.0.1:32860)
0928 16:34:22.473544 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:22.473559 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:22.473651 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:22.474282 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:22.474285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:22.474302 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:22.474400 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.474409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.475635 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.475638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.476878 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.476886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.477874 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.477877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.477989 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.478025 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 16:34:22.478134 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 16:34:22.478258 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":50,"thread_start_us":111,"threads_started":1}
I20260928 16:34:22.478644 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.473237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:34009 (local address 127.0.0.1:58322)
0928 16:34:22.473674 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:22.473687 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:22.473763 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:22.474487 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:22.474490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:22.474512 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:22.474589 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.474596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.476105 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.476108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.477115 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.477126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.478203 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.478206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.478298 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.478362 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 16:34:22.478428 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 16:34:22.478499 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":75,"thread_start_us":74,"threads_started":1}
I20260928 16:34:22.478817 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.472350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.27.122.193:33059)
0928 16:34:22.472643 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.472662 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.472687 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:34:22.473007 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.473013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.473252 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.473419 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:22.473947 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.475094 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.477050 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.477813 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.477953 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.478082 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.478580 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.478679 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":80,"threads_started":1}
I20260928 16:34:22.479055 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.472144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:33059 (local address 127.0.0.1:47082)
0928 16:34:22.472537 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:22.472599 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:22.472731 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:22.473630 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:22.473638 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:22.473662 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:22.473813 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.473821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.475215 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.475219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.476874 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.476886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.478150 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.478156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.478265 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.478367 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 16:34:22.478428 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 16:34:22.478501 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":118,"thread_start_us":98,"threads_started":1}
I20260928 16:34:22.479334 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.473645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58322 (local address 127.27.122.195:34009)
0928 16:34:22.474036 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.474047 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.474069 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:34:22.474156 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.474161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.474350 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.474497 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:22.474734 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.475964 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.477247 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.478081 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.478814 (+ 733us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.478954 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.479016 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.479185 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":101,"threads_started":1}
I20260928 16:34:22.482151 29396 tablet_service.cc:1467] Processing CreateTablet for tablet e987bdba090b46918d1611310bf46786 (DEFAULT_TABLE table=test-table [id=bf50a7705b2143a29adca8b18ffe65d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:22.482151 29536 tablet_service.cc:1467] Processing CreateTablet for tablet e987bdba090b46918d1611310bf46786 (DEFAULT_TABLE table=test-table [id=bf50a7705b2143a29adca8b18ffe65d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:22.483372 29396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e987bdba090b46918d1611310bf46786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:22.483372 29536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e987bdba090b46918d1611310bf46786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:22.484709 29676 tablet_service.cc:1467] Processing CreateTablet for tablet e987bdba090b46918d1611310bf46786 (DEFAULT_TABLE table=test-table [id=bf50a7705b2143a29adca8b18ffe65d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:22.485657 29676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e987bdba090b46918d1611310bf46786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:22.491608 29765 tablet_bootstrap.cc:492] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf: Bootstrap starting.
I20260928 16:34:22.492523 29766 tablet_bootstrap.cc:492] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9: Bootstrap starting.
I20260928 16:34:22.492991 29767 tablet_bootstrap.cc:492] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088: Bootstrap starting.
I20260928 16:34:22.494204 29765 tablet_bootstrap.cc:654] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:22.494683 29766 tablet_bootstrap.cc:654] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:22.494957 29767 tablet_bootstrap.cc:654] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:22.495074 29765 log.cc:824] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:22.495512 29766 log.cc:824] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:22.495791 29767 log.cc:824] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:22.497232 29765 tablet_bootstrap.cc:492] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf: No bootstrap required, opened a new log
I20260928 16:34:22.497455 29765 ts_tablet_manager.cc:1397] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:34:22.497823 29767 tablet_bootstrap.cc:492] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088: No bootstrap required, opened a new log
I20260928 16:34:22.497839 29766 tablet_bootstrap.cc:492] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9: No bootstrap required, opened a new log
I20260928 16:34:22.498015 29766 ts_tablet_manager.cc:1397] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:34:22.498015 29767 ts_tablet_manager.cc:1397] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:34:22.500598 29765 raft_consensus.cc:348] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.500831 29765 raft_consensus.cc:374] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:22.500891 29765 raft_consensus.cc:729] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d392313b38d6412092161eac7a4a16bf, State: Initialized, Role: FOLLOWER
I20260928 16:34:22.501353 29765 consensus_queue.cc:260] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.502007 29742 heartbeater.cc:503] Master 127.27.122.254:34995 was elected leader, sending a full tablet report...
I20260928 16:34:22.502012 29766 raft_consensus.cc:348] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.502310 29766 raft_consensus.cc:374] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:22.502355 29765 ts_tablet_manager.cc:1428] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:34:22.502372 29766 raft_consensus.cc:729] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c486231190c4436ca2a4ec5e5e1ef2e9, State: Initialized, Role: FOLLOWER
I20260928 16:34:22.502800 29766 consensus_queue.cc:260] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.503108 29767 raft_consensus.cc:348] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.503348 29767 raft_consensus.cc:374] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:22.503409 29767 raft_consensus.cc:729] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccad18f5ab7e4f19a4dc53815bf4c088, State: Initialized, Role: FOLLOWER
I20260928 16:34:22.503603 29462 heartbeater.cc:503] Master 127.27.122.254:34995 was elected leader, sending a full tablet report...
I20260928 16:34:22.503811 29767 consensus_queue.cc:260] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.503918 29766 ts_tablet_manager.cc:1428] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 16:34:22.504416 29602 heartbeater.cc:503] Master 127.27.122.254:34995 was elected leader, sending a full tablet report...
I20260928 16:34:22.504767 29767 ts_tablet_manager.cc:1428] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:34:22.544591 29771 raft_consensus.cc:482] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:22.544809 29771 raft_consensus.cc:504] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.546031 29771 leader_election.cc:288] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c486231190c4436ca2a4ec5e5e1ef2e9 (127.27.122.193:33059), ccad18f5ab7e4f19a4dc53815bf4c088 (127.27.122.194:35721)
I20260928 16:34:22.550273 29635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.546243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:33059 (local address 127.27.122.195:45299)
0928 16:34:22.546360 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:22.546374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:22.546464 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:22.546764 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:22.546766 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:22.546784 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:22.546917 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.546923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.547866 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.547870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.549223 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.549235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.549949 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.549952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.550042 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.550063 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:34:22.550122 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 16:34:22.550175 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 16:34:22.550364 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.546354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:45299 (local address 127.27.122.193:33059)
0928 16:34:22.546503 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.546510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.546526 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:34:22.546592 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.546595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.546642 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.546728 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:22.547043 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.547754 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.549347 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.549837 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.549961 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.550012 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.550159 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.550231 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:34:22.550954 29774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.546482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:35721 (local address 127.27.122.195:48461)
0928 16:34:22.546830 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:22.546842 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:22.546918 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:22.547143 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:22.547145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:22.547163 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:22.547284 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.547289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.548086 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.548089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.549586 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.549594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.550544 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.550550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.550649 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.550678 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:34:22.550751 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 16:34:22.550819 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":120,"threads_started":1}
I20260928 16:34:22.550954 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.546772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:48461 (local address 127.27.122.194:35721)
0928 16:34:22.546914 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.546919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.546935 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:34:22.546991 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.546993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.547039 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.547120 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:22.547392 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.547987 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.549698 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.550285 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.550406 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.550459 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.550779 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.550850 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:34:22.550918 29416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e987bdba090b46918d1611310bf46786" candidate_uuid: "d392313b38d6412092161eac7a4a16bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" is_pre_election: true
I20260928 16:34:22.551244 29416 raft_consensus.cc:2454] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d392313b38d6412092161eac7a4a16bf in term 0.
I20260928 16:34:22.551582 29556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e987bdba090b46918d1611310bf46786" candidate_uuid: "d392313b38d6412092161eac7a4a16bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" is_pre_election: true
I20260928 16:34:22.551792 29630 leader_election.cc:302] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c486231190c4436ca2a4ec5e5e1ef2e9, d392313b38d6412092161eac7a4a16bf; no voters:
I20260928 16:34:22.551862 29556 raft_consensus.cc:2454] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d392313b38d6412092161eac7a4a16bf in term 0.
I20260928 16:34:22.552086 29771 raft_consensus.cc:2781] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:22.552181 29771 raft_consensus.cc:482] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:22.552253 29771 raft_consensus.cc:3037] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:22.553387 29771 raft_consensus.cc:504] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.553699 29771 leader_election.cc:288] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [CANDIDATE]: Term 1 election: Requested vote from peers c486231190c4436ca2a4ec5e5e1ef2e9 (127.27.122.193:33059), ccad18f5ab7e4f19a4dc53815bf4c088 (127.27.122.194:35721)
I20260928 16:34:22.554020 29416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e987bdba090b46918d1611310bf46786" candidate_uuid: "d392313b38d6412092161eac7a4a16bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9"
I20260928 16:34:22.554070 29556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e987bdba090b46918d1611310bf46786" candidate_uuid: "d392313b38d6412092161eac7a4a16bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088"
I20260928 16:34:22.554134 29416 raft_consensus.cc:3037] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:22.554244 29556 raft_consensus.cc:3037] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:22.555136 29416 raft_consensus.cc:2454] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d392313b38d6412092161eac7a4a16bf in term 1.
I20260928 16:34:22.555438 29630 leader_election.cc:302] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c486231190c4436ca2a4ec5e5e1ef2e9, d392313b38d6412092161eac7a4a16bf; no voters:
I20260928 16:34:22.555519 29556 raft_consensus.cc:2454] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d392313b38d6412092161eac7a4a16bf in term 1.
I20260928 16:34:22.555624 29771 raft_consensus.cc:2781] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:22.555840 29771 raft_consensus.cc:686] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [term 1 LEADER]: Becoming Leader. State: Replica: d392313b38d6412092161eac7a4a16bf, State: Running, Role: LEADER
I20260928 16:34:22.556159 29771 consensus_queue.cc:237] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.558911 29259 catalog_manager.cc:5495] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf reported cstate change: term changed from 0 to 1, leader changed from <none> to d392313b38d6412092161eac7a4a16bf (127.27.122.195). New cstate: current_term: 1 leader_uuid: "d392313b38d6412092161eac7a4a16bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } health_report { overall_health: HEALTHY } } }
I20260928 16:34:22.578394 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.574712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.27.122.195:34009)
0928 16:34:22.574854 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.574877 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.574895 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:34:22.575086 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.575089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.575142 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.575225 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:22.575578 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.576140 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.577040 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.577272 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.577349 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.577631 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:22.578079 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:22.578164 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.578244 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.578291 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:34:22.584050 29416 raft_consensus.cc:1261] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 1 FOLLOWER]: Refusing update from remote peer d392313b38d6412092161eac7a4a16bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:22.584322 29556 raft_consensus.cc:1261] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 1 FOLLOWER]: Refusing update from remote peer d392313b38d6412092161eac7a4a16bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:22.584820 29771 consensus_queue.cc:1035] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:22.585069 29775 consensus_queue.cc:1035] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:22.593528 29779 mvcc.cc:205] Tried to move back new op lower bound from 7334351923536990208 to 7334351923432427520. Current Snapshot: MvccSnapshot[applied={T|T < 7334351923536990208}]
I20260928 16:34:22.594682 29780 mvcc.cc:205] Tried to move back new op lower bound from 7334351923536990208 to 7334351923432427520. Current Snapshot: MvccSnapshot[applied={T|T < 7334351923536990208}]
I20260928 16:34:22.602959 29258 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:22.603185 29258 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:22.605705 29258 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db: Sending DeleteTablet for 3 replicas of tablet e987bdba090b46918d1611310bf46786
I20260928 16:34:22.606397 29396 tablet_service.cc:1514] Processing DeleteTablet for tablet e987bdba090b46918d1611310bf46786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:22 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:47082
I20260928 16:34:22.606496 29676 tablet_service.cc:1514] Processing DeleteTablet for tablet e987bdba090b46918d1611310bf46786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:22 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:58322
I20260928 16:34:22.606905 29788 tablet_replica.cc:330] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf: stopping tablet replica
I20260928 16:34:22.606894 29536 tablet_service.cc:1514] Processing DeleteTablet for tablet e987bdba090b46918d1611310bf46786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:22 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:32860
I20260928 16:34:22.607167 29259 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:34:22.607338 29789 tablet_replica.cc:330] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088: stopping tablet replica
I20260928 16:34:22.607375 29787 tablet_replica.cc:330] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9: stopping tablet replica
W20260928 16:34:22.607574 29259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:22.607610 29789 raft_consensus.cc:2229] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:22.607666 29787 raft_consensus.cc:2229] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:22.607913 29789 raft_consensus.cc:2258] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:22.607982 29787 raft_consensus.cc:2258] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:22.609439 29787 ts_tablet_manager.cc:1905] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:22.609447 29789 ts_tablet_manager.cc:1905] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:22.610581 29788 raft_consensus.cc:2229] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:22.611450 29788 raft_consensus.cc:2258] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:22.612362 29787 ts_tablet_manager.cc:1918] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:22.612466 29787 log.cc:1196] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/wal/wals/e987bdba090b46918d1611310bf46786
I20260928 16:34:22.612777 29789 ts_tablet_manager.cc:1918] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:22.612824 29787 ts_tablet_manager.cc:1939] T e987bdba090b46918d1611310bf46786 P c486231190c4436ca2a4ec5e5e1ef2e9: Deleting consensus metadata
I20260928 16:34:22.612871 29789 log.cc:1196] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/e987bdba090b46918d1611310bf46786
I20260928 16:34:22.613219 29789 ts_tablet_manager.cc:1939] T e987bdba090b46918d1611310bf46786 P ccad18f5ab7e4f19a4dc53815bf4c088: Deleting consensus metadata
I20260928 16:34:22.613265 29396 tablet_service.cc:1467] Processing CreateTablet for tablet c66fa14b33084dc7a5e8b63f3e701ac1 (DEFAULT_TABLE table=test-table [id=a6d627de88334445b497c46b3b772e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:22.613510 29396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66fa14b33084dc7a5e8b63f3e701ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:22.615124 29788 ts_tablet_manager.cc:1905] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:22.615839 29766 tablet_bootstrap.cc:492] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9: Bootstrap starting.
I20260928 16:34:22.616824 29766 tablet_bootstrap.cc:654] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:22.617653 29246 catalog_manager.cc:4841] TS ccad18f5ab7e4f19a4dc53815bf4c088 (127.27.122.194:35721): tablet e987bdba090b46918d1611310bf46786 (table test-table [id=bf50a7705b2143a29adca8b18ffe65d8]) successfully deleted
I20260928 16:34:22.617841 29246 catalog_manager.cc:4841] TS c486231190c4436ca2a4ec5e5e1ef2e9 (127.27.122.193:33059): tablet e987bdba090b46918d1611310bf46786 (table test-table [id=bf50a7705b2143a29adca8b18ffe65d8]) successfully deleted
I20260928 16:34:22.617893 29536 tablet_service.cc:1467] Processing CreateTablet for tablet c66fa14b33084dc7a5e8b63f3e701ac1 (DEFAULT_TABLE table=test-table [id=a6d627de88334445b497c46b3b772e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:22.617949 29788 ts_tablet_manager.cc:1918] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:22.618031 29788 log.cc:1196] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/e987bdba090b46918d1611310bf46786
I20260928 16:34:22.618111 29766 tablet_bootstrap.cc:492] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9: No bootstrap required, opened a new log
I20260928 16:34:22.618146 29536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66fa14b33084dc7a5e8b63f3e701ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:22.618185 29766 ts_tablet_manager.cc:1397] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:22.618338 29788 ts_tablet_manager.cc:1939] T e987bdba090b46918d1611310bf46786 P d392313b38d6412092161eac7a4a16bf: Deleting consensus metadata
I20260928 16:34:22.618419 29676 tablet_service.cc:1467] Processing CreateTablet for tablet c66fa14b33084dc7a5e8b63f3e701ac1 (DEFAULT_TABLE table=test-table [id=a6d627de88334445b497c46b3b772e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:22.618728 29676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66fa14b33084dc7a5e8b63f3e701ac1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:22.618920 29766 raft_consensus.cc:348] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.619050 29766 raft_consensus.cc:374] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:22.619084 29766 raft_consensus.cc:729] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c486231190c4436ca2a4ec5e5e1ef2e9, State: Initialized, Role: FOLLOWER
I20260928 16:34:22.619259 29766 consensus_queue.cc:260] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.620048 29766 ts_tablet_manager.cc:1428] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:22.620369 29767 tablet_bootstrap.cc:492] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088: Bootstrap starting.
I20260928 16:34:22.621379 29767 tablet_bootstrap.cc:654] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:22.621455 29765 tablet_bootstrap.cc:492] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf: Bootstrap starting.
I20260928 16:34:22.621460 29246 catalog_manager.cc:4841] TS d392313b38d6412092161eac7a4a16bf (127.27.122.195:34009): tablet e987bdba090b46918d1611310bf46786 (table test-table [id=bf50a7705b2143a29adca8b18ffe65d8]) successfully deleted
I20260928 16:34:22.622514 29765 tablet_bootstrap.cc:654] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:22.623665 29765 tablet_bootstrap.cc:492] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf: No bootstrap required, opened a new log
I20260928 16:34:22.623669 29767 tablet_bootstrap.cc:492] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088: No bootstrap required, opened a new log
I20260928 16:34:22.623739 29767 ts_tablet_manager.cc:1397] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:34:22.623772 29765 ts_tablet_manager.cc:1397] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:22.624151 29765 raft_consensus.cc:348] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.624177 29767 raft_consensus.cc:348] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.624264 29765 raft_consensus.cc:374] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:22.624310 29765 raft_consensus.cc:729] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d392313b38d6412092161eac7a4a16bf, State: Initialized, Role: FOLLOWER
I20260928 16:34:22.624315 29767 raft_consensus.cc:374] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:22.624380 29767 raft_consensus.cc:729] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccad18f5ab7e4f19a4dc53815bf4c088, State: Initialized, Role: FOLLOWER
I20260928 16:34:22.624439 29765 consensus_queue.cc:260] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.624541 29767 consensus_queue.cc:260] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.624727 29765 ts_tablet_manager.cc:1428] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:22.625211 29767 ts_tablet_manager.cc:1428] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260928 16:34:22.667932 29743 tablet.cc:2369] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:34:22.700632 29603 tablet.cc:2369] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:34:22.744604 29463 tablet.cc:2369] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:22.874549 29773 raft_consensus.cc:482] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:22.874722 29773 raft_consensus.cc:504] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.875986 29773 leader_election.cc:288] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c486231190c4436ca2a4ec5e5e1ef2e9 (127.27.122.193:33059), d392313b38d6412092161eac7a4a16bf (127.27.122.195:34009)
I20260928 16:34:22.880429 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.876411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:60805 (local address 127.27.122.193:33059)
0928 16:34:22.876598 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.876605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.876662 (+ 57us) server_negotiation.cc:407] Connection header received
0928 16:34:22.876927 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.876931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.876986 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.877082 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:22.877447 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.878166 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.879357 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.879803 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.879896 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.879937 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.880220 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.880301 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:34:22.880441 29793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.876285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:33059 (local address 127.27.122.194:60805)
0928 16:34:22.876583 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:22.876597 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:22.876733 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:22.877110 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:22.877114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:22.877162 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:22.877314 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.877324 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.878253 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.878257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.879189 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.879199 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.879930 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.879934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.880041 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.880106 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 16:34:22.880190 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:34:22.880276 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":47,"thread_start_us":104,"threads_started":1}
I20260928 16:34:22.880476 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.876870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:51121 (local address 127.27.122.195:34009)
0928 16:34:22.877031 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.877053 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.877074 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:34:22.877217 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.877220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.877286 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.877372 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:22.877690 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.878297 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.879501 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.879922 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.880039 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.880099 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.880294 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.880370 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:34:22.880451 29794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.876730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:34009 (local address 127.27.122.194:51121)
0928 16:34:22.877022 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:22.877037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:22.877108 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:22.877422 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:22.877427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:22.877445 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:22.877548 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.877553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.878406 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.878408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.879378 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.879388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.880044 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.880047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.880140 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.880179 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 16:34:22.880258 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 16:34:22.880320 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":5,"thread_start_us":56,"threads_started":1}
I20260928 16:34:22.881040 29416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66fa14b33084dc7a5e8b63f3e701ac1" candidate_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" is_pre_election: true
I20260928 16:34:22.881171 29416 raft_consensus.cc:2454] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccad18f5ab7e4f19a4dc53815bf4c088 in term 0.
I20260928 16:34:22.881527 29490 leader_election.cc:302] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c486231190c4436ca2a4ec5e5e1ef2e9, ccad18f5ab7e4f19a4dc53815bf4c088; no voters:
I20260928 16:34:22.881652 29696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66fa14b33084dc7a5e8b63f3e701ac1" candidate_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d392313b38d6412092161eac7a4a16bf" is_pre_election: true
I20260928 16:34:22.881807 29773 raft_consensus.cc:2781] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:22.881866 29773 raft_consensus.cc:482] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:22.881912 29773 raft_consensus.cc:3037] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:22.881906 29696 raft_consensus.cc:2454] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccad18f5ab7e4f19a4dc53815bf4c088 in term 0.
I20260928 16:34:22.883159 29773 raft_consensus.cc:504] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.883524 29773 leader_election.cc:288] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [CANDIDATE]: Term 1 election: Requested vote from peers c486231190c4436ca2a4ec5e5e1ef2e9 (127.27.122.193:33059), d392313b38d6412092161eac7a4a16bf (127.27.122.195:34009)
I20260928 16:34:22.883868 29416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66fa14b33084dc7a5e8b63f3e701ac1" candidate_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9"
I20260928 16:34:22.883914 29696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66fa14b33084dc7a5e8b63f3e701ac1" candidate_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d392313b38d6412092161eac7a4a16bf"
I20260928 16:34:22.884006 29416 raft_consensus.cc:3037] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:22.884074 29696 raft_consensus.cc:3037] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:22.884990 29416 raft_consensus.cc:2454] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccad18f5ab7e4f19a4dc53815bf4c088 in term 1.
I20260928 16:34:22.885253 29696 raft_consensus.cc:2454] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccad18f5ab7e4f19a4dc53815bf4c088 in term 1.
I20260928 16:34:22.885296 29490 leader_election.cc:302] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c486231190c4436ca2a4ec5e5e1ef2e9, ccad18f5ab7e4f19a4dc53815bf4c088; no voters:
I20260928 16:34:22.885468 29773 raft_consensus.cc:2781] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:22.885617 29773 raft_consensus.cc:686] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 1 LEADER]: Becoming Leader. State: Replica: ccad18f5ab7e4f19a4dc53815bf4c088, State: Running, Role: LEADER
I20260928 16:34:22.885887 29773 consensus_queue.cc:237] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.888056 29258 catalog_manager.cc:5495] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 reported cstate change: term changed from 0 to 1, leader changed from <none> to ccad18f5ab7e4f19a4dc53815bf4c088 (127.27.122.194). New cstate: current_term: 1 leader_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } health_report { overall_health: UNKNOWN } } }
Sep 28 16:34:22 dist-test-slave-g9sh krb5kdc[29214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613261, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:22.935171 29802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.926588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:34995 (local address 127.0.0.1:44636)
0928 16:34:22.926892 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:22.926909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:22.927022 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:22.927262 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:22.927264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:22.927697 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:22.927795 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.927801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.928529 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.928531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.929159 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:22.929167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.929556 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.929558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:22.929618 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.930218 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:22.930231 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:22.931815 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:22.933734 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:22.933743 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:22.933748 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:22.933973 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:22.934266 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:22.934268 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:22.934270 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:22.934346 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:22.934724 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:22.934730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:22.934828 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 16:34:22.934917 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:34:22.935032 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":130,"threads_started":1}
I20260928 16:34:22.935287 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:22.926679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.27.122.254:34995)
0928 16:34:22.926802 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:22.926806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:22.926987 (+ 181us) server_negotiation.cc:407] Connection header received
0928 16:34:22.927103 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:22.927105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:22.927145 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:22.927221 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:22.927900 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.928421 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.929258 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:22.929459 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:22.929538 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:22.931946 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:22.931949 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:22.931950 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:22.931972 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:22.933639 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.934072 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.934075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.934076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.934167 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:22.934420 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:22.934422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:22.934423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:22.934632 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:22.934710 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:22.934972 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 16:34:22.935090 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":45}
I20260928 16:34:22.939009 29258 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:44636:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:34:22.939765 29258 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:22.944411 29536 tablet_service.cc:1467] Processing CreateTablet for tablet 108b142ea7524aa89e87a7d8a697b22d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0be7f037fbf64e979492caad534efcbb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:34:22.944387 29396 tablet_service.cc:1467] Processing CreateTablet for tablet 108b142ea7524aa89e87a7d8a697b22d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0be7f037fbf64e979492caad534efcbb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:34:22.944796 29536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108b142ea7524aa89e87a7d8a697b22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:22.944796 29396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108b142ea7524aa89e87a7d8a697b22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:22.944655 29676 tablet_service.cc:1467] Processing CreateTablet for tablet 108b142ea7524aa89e87a7d8a697b22d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0be7f037fbf64e979492caad534efcbb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:34:22.945055 29676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108b142ea7524aa89e87a7d8a697b22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:22.947552 29765 tablet_bootstrap.cc:492] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf: Bootstrap starting.
I20260928 16:34:22.947674 29767 tablet_bootstrap.cc:492] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088: Bootstrap starting.
I20260928 16:34:22.947719 29766 tablet_bootstrap.cc:492] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9: Bootstrap starting.
I20260928 16:34:22.948374 29765 tablet_bootstrap.cc:654] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:22.948813 29766 tablet_bootstrap.cc:654] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:22.948848 29767 tablet_bootstrap.cc:654] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:22.949635 29765 tablet_bootstrap.cc:492] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf: No bootstrap required, opened a new log
I20260928 16:34:22.949736 29765 ts_tablet_manager.cc:1397] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 16:34:22.950290 29766 tablet_bootstrap.cc:492] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9: No bootstrap required, opened a new log
I20260928 16:34:22.950229 29765 raft_consensus.cc:348] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.950341 29765 raft_consensus.cc:374] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:22.950377 29765 raft_consensus.cc:729] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d392313b38d6412092161eac7a4a16bf, State: Initialized, Role: FOLLOWER
I20260928 16:34:22.950394 29766 ts_tablet_manager.cc:1397] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:34:22.950476 29767 tablet_bootstrap.cc:492] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088: No bootstrap required, opened a new log
I20260928 16:34:22.950553 29767 ts_tablet_manager.cc:1397] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:34:22.950522 29765 consensus_queue.cc:260] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.950800 29765 ts_tablet_manager.cc:1428] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:22.950907 29766 raft_consensus.cc:348] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.951042 29766 raft_consensus.cc:374] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:22.951077 29766 raft_consensus.cc:729] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c486231190c4436ca2a4ec5e5e1ef2e9, State: Initialized, Role: FOLLOWER
I20260928 16:34:22.951030 29767 raft_consensus.cc:348] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.951134 29767 raft_consensus.cc:374] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:22.951074 29771 tablet_replica.cc:439] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } } }
I20260928 16:34:22.951166 29767 raft_consensus.cc:729] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccad18f5ab7e4f19a4dc53815bf4c088, State: Initialized, Role: FOLLOWER
I20260928 16:34:22.951203 29771 tablet_replica.cc:442] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:22.951186 29766 consensus_queue.cc:260] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.951498 29766 ts_tablet_manager.cc:1428] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:22.951467 29772 tablet_replica.cc:439] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } } }
I20260928 16:34:22.951493 29767 consensus_queue.cc:260] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.951604 29772 tablet_replica.cc:442] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:22.951772 29767 ts_tablet_manager.cc:1428] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:22.951740 29773 tablet_replica.cc:439] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } } }
I20260928 16:34:22.951856 29773 tablet_replica.cc:442] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:22.987097 29771 raft_consensus.cc:482] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:22.987263 29771 raft_consensus.cc:504] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.987792 29771 leader_election.cc:288] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c486231190c4436ca2a4ec5e5e1ef2e9 (127.27.122.193:33059), ccad18f5ab7e4f19a4dc53815bf4c088 (127.27.122.194:35721)
I20260928 16:34:22.988201 29416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108b142ea7524aa89e87a7d8a697b22d" candidate_uuid: "d392313b38d6412092161eac7a4a16bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" is_pre_election: true
I20260928 16:34:22.988255 29556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108b142ea7524aa89e87a7d8a697b22d" candidate_uuid: "d392313b38d6412092161eac7a4a16bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" is_pre_election: true
I20260928 16:34:22.988379 29416 raft_consensus.cc:2454] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d392313b38d6412092161eac7a4a16bf in term 0.
I20260928 16:34:22.988415 29556 raft_consensus.cc:2454] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d392313b38d6412092161eac7a4a16bf in term 0.
I20260928 16:34:22.988718 29630 leader_election.cc:302] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccad18f5ab7e4f19a4dc53815bf4c088, d392313b38d6412092161eac7a4a16bf; no voters:
I20260928 16:34:22.988933 29771 raft_consensus.cc:2781] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:22.988998 29771 raft_consensus.cc:482] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:22.989040 29771 raft_consensus.cc:3037] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:22.990022 29771 raft_consensus.cc:504] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.990339 29771 leader_election.cc:288] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [CANDIDATE]: Term 1 election: Requested vote from peers c486231190c4436ca2a4ec5e5e1ef2e9 (127.27.122.193:33059), ccad18f5ab7e4f19a4dc53815bf4c088 (127.27.122.194:35721)
I20260928 16:34:22.990653 29416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108b142ea7524aa89e87a7d8a697b22d" candidate_uuid: "d392313b38d6412092161eac7a4a16bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9"
I20260928 16:34:22.990710 29556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108b142ea7524aa89e87a7d8a697b22d" candidate_uuid: "d392313b38d6412092161eac7a4a16bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088"
I20260928 16:34:22.990756 29416 raft_consensus.cc:3037] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:22.990849 29556 raft_consensus.cc:3037] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:22.991712 29416 raft_consensus.cc:2454] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d392313b38d6412092161eac7a4a16bf in term 1.
I20260928 16:34:22.991825 29556 raft_consensus.cc:2454] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d392313b38d6412092161eac7a4a16bf in term 1.
I20260928 16:34:22.991987 29630 leader_election.cc:302] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c486231190c4436ca2a4ec5e5e1ef2e9, d392313b38d6412092161eac7a4a16bf; no voters:
I20260928 16:34:22.992223 29771 raft_consensus.cc:2781] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:22.992317 29771 raft_consensus.cc:686] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [term 1 LEADER]: Becoming Leader. State: Replica: d392313b38d6412092161eac7a4a16bf, State: Running, Role: LEADER
I20260928 16:34:22.992478 29771 consensus_queue.cc:237] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } }
I20260928 16:34:22.993170 29775 tablet_replica.cc:439] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf: TxnStatusTablet state changed. Reason: New leader d392313b38d6412092161eac7a4a16bf. Latest consensus state: current_term: 1 leader_uuid: "d392313b38d6412092161eac7a4a16bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } } }
I20260928 16:34:22.993283 29775 tablet_replica.cc:442] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:34:22.994150 29259 catalog_manager.cc:5495] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf reported cstate change: term changed from 0 to 1, leader changed from <none> to d392313b38d6412092161eac7a4a16bf (127.27.122.195). New cstate: current_term: 1 leader_uuid: "d392313b38d6412092161eac7a4a16bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } health_report { overall_health: HEALTHY } } }
I20260928 16:34:22.997057 29807 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:34:22.997224 29807 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:34:23.006779 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:23.003410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58346 (local address 127.27.122.195:34009)
0928 16:34:23.003509 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:23.003514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:23.003534 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:34:23.003572 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:23.003574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:23.003616 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:23.003707 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:23.003984 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.004573 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.005522 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.005817 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.005929 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:23.006248 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:23.006340 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:23.006424 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:23.006575 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 16:34:23.006634 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 16:34:23.006779 29802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:23.003167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:34009 (local address 127.0.0.1:58346)
0928 16:34:23.003320 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:23.003337 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:23.003439 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:23.003732 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:23.003735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:23.003743 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:34:23.003874 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:23.003879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.004683 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.004687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:23.005395 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:23.005405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.005929 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.005933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:23.006053 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:23.006109 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:34:23.006470 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:34:23.006485 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 16:34:23.006561 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 16:34:23.006654 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 16:34:23.009270 29556 raft_consensus.cc:1261] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088 [term 1 FOLLOWER]: Refusing update from remote peer d392313b38d6412092161eac7a4a16bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:23.009270 29416 raft_consensus.cc:1261] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9 [term 1 FOLLOWER]: Refusing update from remote peer d392313b38d6412092161eac7a4a16bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:23.009716 29775 consensus_queue.cc:1035] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:23.009867 29771 consensus_queue.cc:1035] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:23.011224 29772 tablet_replica.cc:439] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9: TxnStatusTablet state changed. Reason: New leader d392313b38d6412092161eac7a4a16bf. Latest consensus state: current_term: 1 leader_uuid: "d392313b38d6412092161eac7a4a16bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } } }
I20260928 16:34:23.011348 29772 tablet_replica.cc:442] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:23.011439 29773 tablet_replica.cc:439] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088: TxnStatusTablet state changed. Reason: New leader d392313b38d6412092161eac7a4a16bf. Latest consensus state: current_term: 1 leader_uuid: "d392313b38d6412092161eac7a4a16bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } } }
I20260928 16:34:23.011562 29773 tablet_replica.cc:442] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:23.012876 29775 tablet_replica.cc:439] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d392313b38d6412092161eac7a4a16bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } } }
I20260928 16:34:23.013002 29775 tablet_replica.cc:442] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:34:23.013221 29778 mvcc.cc:205] Tried to move back new op lower bound from 7334351925281419264 to 7334351925218545664. Current Snapshot: MvccSnapshot[applied={T|T < 7334351925281419264}]
I20260928 16:34:23.015066 29772 tablet_replica.cc:439] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d392313b38d6412092161eac7a4a16bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } } }
I20260928 16:34:23.015172 29772 tablet_replica.cc:442] T 108b142ea7524aa89e87a7d8a697b22d P c486231190c4436ca2a4ec5e5e1ef2e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:23.022068 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:23.018434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.27.122.194:35721)
0928 16:34:23.018561 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:23.018566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:23.018588 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:34:23.018743 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:23.018746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:23.018809 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:23.018920 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:23.019182 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.019684 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.020517 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.020761 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.020874 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:23.021190 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:23.021682 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:23.021803 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:23.021873 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 16:34:23.021925 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:34:23.022167 29771 tablet_replica.cc:439] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d392313b38d6412092161eac7a4a16bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } } }
I20260928 16:34:23.022282 29771 tablet_replica.cc:442] T 108b142ea7524aa89e87a7d8a697b22d P d392313b38d6412092161eac7a4a16bf: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:34:23.022737 29773 tablet_replica.cc:439] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d392313b38d6412092161eac7a4a16bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 } } peers { permanent_uuid: "ccad18f5ab7e4f19a4dc53815bf4c088" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35721 } } peers { permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 } } }
I20260928 16:34:23.022845 29773 tablet_replica.cc:442] T 108b142ea7524aa89e87a7d8a697b22d P ccad18f5ab7e4f19a4dc53815bf4c088: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:23.033823 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:23.029793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:38831 (local address 127.27.122.195:34009)
0928 16:34:23.029906 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:23.029909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:23.030130 (+ 221us) server_negotiation.cc:407] Connection header received
0928 16:34:23.030247 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:23.030249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:23.030288 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:23.030347 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:23.030603 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.031052 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.032257 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.032454 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.032558 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:23.033366 (+ 808us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:23.033431 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:23.033489 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:23.033673 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 16:34:23.033731 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 16:34:23.033852 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:23.029695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:34009 (local address 127.27.122.194:38831)
0928 16:34:23.030047 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:23.030062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:23.030162 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:23.030379 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:23.030382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:23.030389 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:34:23.030505 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:23.030511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.031156 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.031159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:23.032143 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:23.032153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.033077 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.033082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:23.033181 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:23.033232 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:34:23.033518 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:34:23.033524 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:34:23.033617 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 16:34:23.033692 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":99,"threads_started":1}
I20260928 16:34:23.040742 29416 raft_consensus.cc:1261] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 1 FOLLOWER]: Refusing update from remote peer ccad18f5ab7e4f19a4dc53815bf4c088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:23.040985 29696 raft_consensus.cc:1261] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf [term 1 FOLLOWER]: Refusing update from remote peer ccad18f5ab7e4f19a4dc53815bf4c088: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:23.041575 29773 consensus_queue.cc:1035] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c486231190c4436ca2a4ec5e5e1ef2e9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 33059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:23.041790 29795 consensus_queue.cc:1035] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d392313b38d6412092161eac7a4a16bf" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:23.064030 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:23.060408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:53881 (local address 127.27.122.194:35721)
0928 16:34:23.060538 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:23.060543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:23.060761 (+ 218us) server_negotiation.cc:407] Connection header received
0928 16:34:23.060859 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:23.060861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:23.060943 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:23.061021 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:23.061244 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.061720 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.062899 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.063129 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.063208 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:23.063636 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:23.063720 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:23.063794 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:23.063880 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 16:34:23.063933 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:34:23.064111 29819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:23.060344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:35721 (local address 127.27.122.195:53881)
0928 16:34:23.060668 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:23.060683 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:23.060784 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:23.061042 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:23.061045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:23.061053 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:34:23.061158 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:23.061163 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.061817 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.061821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:23.062787 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:23.062798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.063251 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.063255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:23.063344 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:23.063395 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:34:23.063798 (+ 403us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:34:23.063805 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 16:34:23.063870 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 16:34:23.063951 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":125,"threads_started":1}
I20260928 16:34:23.086604 29258 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:23.086776 29258 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:23.088989 29258 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 384ca5ebebbd4f8088298f06813c43db: Sending DeleteTablet for 3 replicas of tablet c66fa14b33084dc7a5e8b63f3e701ac1
I20260928 16:34:23.089643 29396 tablet_service.cc:1514] Processing DeleteTablet for tablet c66fa14b33084dc7a5e8b63f3e701ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:23 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:47082
I20260928 16:34:23.089834 29787 tablet_replica.cc:330] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9: stopping tablet replica
I20260928 16:34:23.089890 29676 tablet_service.cc:1514] Processing DeleteTablet for tablet c66fa14b33084dc7a5e8b63f3e701ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:23 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:58322
I20260928 16:34:23.089973 29787 raft_consensus.cc:2229] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:23.090013 29536 tablet_service.cc:1514] Processing DeleteTablet for tablet c66fa14b33084dc7a5e8b63f3e701ac1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:23 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:32860
I20260928 16:34:23.090104 29787 raft_consensus.cc:2258] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:23.090188 29789 tablet_replica.cc:330] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088: stopping tablet replica
I20260928 16:34:23.090286 29788 tablet_replica.cc:330] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf: stopping tablet replica
I20260928 16:34:23.090328 29789 raft_consensus.cc:2229] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:23.090394 29788 raft_consensus.cc:2229] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:23.090493 29788 raft_consensus.cc:2258] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:23.090595 29789 raft_consensus.cc:2258] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:23.091264 29789 ts_tablet_manager.cc:1905] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:23.091275 29787 ts_tablet_manager.cc:1905] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:23.093796 29787 ts_tablet_manager.cc:1918] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:23.093891 29787 log.cc:1196] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-0/wal/wals/c66fa14b33084dc7a5e8b63f3e701ac1
I20260928 16:34:23.094043 29789 ts_tablet_manager.cc:1918] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:23.094122 29789 log.cc:1196] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/c66fa14b33084dc7a5e8b63f3e701ac1
I20260928 16:34:23.094174 29787 ts_tablet_manager.cc:1939] T c66fa14b33084dc7a5e8b63f3e701ac1 P c486231190c4436ca2a4ec5e5e1ef2e9: Deleting consensus metadata
I20260928 16:34:23.094377 29789 ts_tablet_manager.cc:1939] T c66fa14b33084dc7a5e8b63f3e701ac1 P ccad18f5ab7e4f19a4dc53815bf4c088: Deleting consensus metadata
I20260928 16:34:23.094871 29246 catalog_manager.cc:4841] TS c486231190c4436ca2a4ec5e5e1ef2e9 (127.27.122.193:33059): tablet c66fa14b33084dc7a5e8b63f3e701ac1 (table test-table [id=a6d627de88334445b497c46b3b772e94]) successfully deleted
I20260928 16:34:23.095163 29246 catalog_manager.cc:4841] TS ccad18f5ab7e4f19a4dc53815bf4c088 (127.27.122.194:35721): tablet c66fa14b33084dc7a5e8b63f3e701ac1 (table test-table [id=a6d627de88334445b497c46b3b772e94]) successfully deleted
I20260928 16:34:23.095412 29788 ts_tablet_manager.cc:1905] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:23.097977 29788 ts_tablet_manager.cc:1918] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:23.098055 29788 log.cc:1196] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/c66fa14b33084dc7a5e8b63f3e701ac1
I20260928 16:34:23.098299 29788 ts_tablet_manager.cc:1939] T c66fa14b33084dc7a5e8b63f3e701ac1 P d392313b38d6412092161eac7a4a16bf: Deleting consensus metadata
I20260928 16:34:23.098937 29246 catalog_manager.cc:4841] TS d392313b38d6412092161eac7a4a16bf (127.27.122.195:34009): tablet c66fa14b33084dc7a5e8b63f3e701ac1 (table test-table [id=a6d627de88334445b497c46b3b772e94]) successfully deleted
Sep 28 16:34:23 dist-test-slave-g9sh krb5kdc[29214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613262, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.122.193@KRBTEST.COM
I20260928 16:34:23.112885 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:23.091805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.27.122.193:33059)
0928 16:34:23.091916 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:23.091920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:23.101238 (+ 9318us) server_negotiation.cc:407] Connection header received
0928 16:34:23.101364 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:23.101368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:23.101428 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:23.101519 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:23.102337 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.103211 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.104531 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.104927 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.105033 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:23.108692 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:23.108706 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:23.108723 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:23.108752 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:23.110872 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:23.111370 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:23.111377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:23.111379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:23.111491 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:23.111800 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:23.111803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:23.111805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:23.112119 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:23.112327 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:23.112561 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 16:34:23.112697 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 16:34:23.113689 29456 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47090
I20260928 16:34:23.124600 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:23.115508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44644 (local address 127.27.122.254:34995)
0928 16:34:23.115658 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:23.115663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:23.115931 (+ 268us) server_negotiation.cc:407] Connection header received
0928 16:34:23.116024 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:23.116027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:23.116082 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:23.116171 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:23.116996 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.117774 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.118919 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:23.119295 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:23.119402 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:23.121046 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:23.121050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:23.121053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:23.121082 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:23.122805 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:23.123277 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:23.123280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:23.123282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:23.123371 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:23.123661 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:23.123665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:23.123666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:23.123892 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:23.124017 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:23.124276 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 16:34:23.124397 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 16:34:23.125382 29258 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44644
I20260928 16:34:23.127036 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 29331
I20260928 16:34:23.136251 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 29471
I20260928 16:34:23.143167 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 29611
I20260928 16:34:23.152273 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 29227
2026-09-28T16:34:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2793 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:23 dist-test-slave-g9sh krb5kdc[29829](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:23 dist-test-slave-g9sh krb5kdc[29829](info): set up 2 sockets
Sep 28 16:34:23 dist-test-slave-g9sh krb5kdc[29829](info): commencing operation
krb5kdc: starting...
W20260928 16:34:24.207258 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:24 dist-test-slave-g9sh krb5kdc[29829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:34:24.328644 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:39421
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:39421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:24.436601 29842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:24.436866 29842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:24.436969 29842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:24.440889 29842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:24.440985 29842 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:34:24.441016 29842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:24.441057 29842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:24.441099 29842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:24.441134 29842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:24.446213 29842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:39421
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:39421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:24.447399 29842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:24.448374 29842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:24.457008 29847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:24.457074 29848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:24.457090 29842 server_base.cc:1034] running on GCE node
W20260928 16:34:24.457007 29850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:24.457917 29842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:24.458952 29842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:24.460140 29842 hybrid_clock.cc:648] HybridClock initialized: now 1790613264460131 us; error 29 us; skew 500 ppm
Sep 28 16:34:24 dist-test-slave-g9sh krb5kdc[29829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613264, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:24.462738 29842 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:34:24.463584 29842 webserver.cc:460] Webserver started at http://127.27.122.254:34403/ using document root <none> and password file <none>
I20260928 16:34:24.464072 29842 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:24.464128 29842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:24.464284 29842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:24.466212 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "cadd6bb2214c4ab4bb3fb10c5aef7ed6"
format_stamp: "Formatted at 2026-09-28 16:34:24 on dist-test-slave-g9sh"
server_key: "5bab9d2b7d77aafd4b206b56e6868689"
server_key_iv: "12b992ae6ad7cec169f7bffffe45cb68"
server_key_version: "encryptionkey@0"
I20260928 16:34:24.466696 29842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "cadd6bb2214c4ab4bb3fb10c5aef7ed6"
format_stamp: "Formatted at 2026-09-28 16:34:24 on dist-test-slave-g9sh"
server_key: "5bab9d2b7d77aafd4b206b56e6868689"
server_key_iv: "12b992ae6ad7cec169f7bffffe45cb68"
server_key_version: "encryptionkey@0"
I20260928 16:34:24.470116 29842 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:34:24.472280 29857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:24.473471 29842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:24.473599 29842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "cadd6bb2214c4ab4bb3fb10c5aef7ed6"
format_stamp: "Formatted at 2026-09-28 16:34:24 on dist-test-slave-g9sh"
server_key: "5bab9d2b7d77aafd4b206b56e6868689"
server_key_iv: "12b992ae6ad7cec169f7bffffe45cb68"
server_key_version: "encryptionkey@0"
I20260928 16:34:24.473694 29842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:24.489236 29842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:24.491819 29842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:24.492019 29842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:24.500038 29842 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:39421
I20260928 16:34:24.500056 29919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:39421 every 8 connection(s)
I20260928 16:34:24.501044 29842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:24.503764 29920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:24.503947 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 29842
I20260928 16:34:24.504082 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:24.504364 28139 external_mini_cluster.cc:1428] Setting key 7181b701575d80d7610a417cccacaca3
I20260928 16:34:24.510954 29920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6: Bootstrap starting.
I20260928 16:34:24.513545 29920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6: Neither blocks nor log segments found. Creating new log.
Sep 28 16:34:24 dist-test-slave-g9sh krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:24.514374 29920 log.cc:824] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:24.516469 29920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6: No bootstrap required, opened a new log
I20260928 16:34:24.519634 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:24.505793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34006 (local address 127.27.122.254:39421)
0928 16:34:24.506221 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:24.506240 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:24.506283 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:34:24.507088 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:24.507111 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:24.507526 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:24.507905 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:24.508527 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.510144 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.511002 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.511678 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.511833 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:24.514325 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:24.514331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:24.514348 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:24.514379 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:24.516859 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:24.517386 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:24.517390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:24.517394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:24.517505 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:24.517795 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:24.517797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:24.517798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:24.518164 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:24.518312 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:24.518613 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 16:34:24.518914 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":116,"threads_started":1}
I20260928 16:34:24.520037 29920 raft_consensus.cc:348] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd6bb2214c4ab4bb3fb10c5aef7ed6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 39421 } }
I20260928 16:34:24.520278 29920 raft_consensus.cc:374] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:24.520366 29920 raft_consensus.cc:729] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadd6bb2214c4ab4bb3fb10c5aef7ed6, State: Initialized, Role: FOLLOWER
I20260928 16:34:24.520850 29920 consensus_queue.cc:260] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd6bb2214c4ab4bb3fb10c5aef7ed6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 39421 } }
I20260928 16:34:24.521026 29920 raft_consensus.cc:388] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:24.521106 29920 raft_consensus.cc:482] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:24.521201 29920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:24.522131 29920 raft_consensus.cc:504] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd6bb2214c4ab4bb3fb10c5aef7ed6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 39421 } }
I20260928 16:34:24.522410 29920 leader_election.cc:302] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cadd6bb2214c4ab4bb3fb10c5aef7ed6; no voters:
I20260928 16:34:24.522648 29920 leader_election.cc:288] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:24.522771 29925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:24.523012 29925 raft_consensus.cc:686] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [term 1 LEADER]: Becoming Leader. State: Replica: cadd6bb2214c4ab4bb3fb10c5aef7ed6, State: Running, Role: LEADER
I20260928 16:34:24.523315 29925 consensus_queue.cc:237] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd6bb2214c4ab4bb3fb10c5aef7ed6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 39421 } }
I20260928 16:34:24.523921 29920 sys_catalog.cc:564] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:24.525179 29927 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cadd6bb2214c4ab4bb3fb10c5aef7ed6. Latest consensus state: current_term: 1 leader_uuid: "cadd6bb2214c4ab4bb3fb10c5aef7ed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd6bb2214c4ab4bb3fb10c5aef7ed6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 39421 } } }
I20260928 16:34:24.525180 29926 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cadd6bb2214c4ab4bb3fb10c5aef7ed6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadd6bb2214c4ab4bb3fb10c5aef7ed6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 39421 } } }
I20260928 16:34:24.525383 29927 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:24.525424 29926 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:24.525830 29936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:24.528829 29936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:24.534524 29936 catalog_manager.cc:1347] Generated new cluster ID: 9f46a67e470c4a949012ecad70901e01
I20260928 16:34:24.534582 29936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:24.543565 29936 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:24.544715 29936 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:24.551189 29936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6: Generated new TSK 0
I20260928 16:34:24.551898 29936 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:34:24.586098 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:39421
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:24.694693 29946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:24.694913 29946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:24.694957 29946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:24.698839 29946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:24.698884 29946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:24.698921 29946 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:34:24.698939 29946 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:34:24.698969 29946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:24.703945 29946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.122.254:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:24.704788 29946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:24.705708 29946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:24.714977 29952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:24.714937 29951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:24.714936 29954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:24.715196 29946 server_base.cc:1034] running on GCE node
I20260928 16:34:24.715627 29946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:24.716238 29946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:24.717392 29946 hybrid_clock.cc:648] HybridClock initialized: now 1790613264717362 us; error 41 us; skew 500 ppm
Sep 28 16:34:24 dist-test-slave-g9sh krb5kdc[29829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613264, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:24.719993 29946 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:34:24.720847 29946 webserver.cc:460] Webserver started at http://127.27.122.193:43015/ using document root <none> and password file <none>
I20260928 16:34:24.721410 29946 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:24.721503 29946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:24.721732 29946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:24.723680 29946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "00e132429f6d4c5690730ec900474d9c"
format_stamp: "Formatted at 2026-09-28 16:34:24 on dist-test-slave-g9sh"
server_key: "01cba4b8fa29d945112b24676f8eac5d"
server_key_iv: "7c4e889d6f192929147b30400e0fea83"
server_key_version: "encryptionkey@0"
I20260928 16:34:24.724366 29946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "00e132429f6d4c5690730ec900474d9c"
format_stamp: "Formatted at 2026-09-28 16:34:24 on dist-test-slave-g9sh"
server_key: "01cba4b8fa29d945112b24676f8eac5d"
server_key_iv: "7c4e889d6f192929147b30400e0fea83"
server_key_version: "encryptionkey@0"
I20260928 16:34:24.728077 29946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:34:24.730494 29961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:24.731637 29946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:34:24.731770 29946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "00e132429f6d4c5690730ec900474d9c"
format_stamp: "Formatted at 2026-09-28 16:34:24 on dist-test-slave-g9sh"
server_key: "01cba4b8fa29d945112b24676f8eac5d"
server_key_iv: "7c4e889d6f192929147b30400e0fea83"
server_key_version: "encryptionkey@0"
I20260928 16:34:24.731880 29946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:24.744633 29946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:24.747315 29946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:24.747512 29946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:24.749011 29946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:24.749087 29946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:24.749154 29946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:24.749189 29946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:34:24 dist-test-slave-g9sh krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613264, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:24.759888 29946 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:40857
I20260928 16:34:24.759965 30077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:40857 every 8 connection(s)
I20260928 16:34:24.760998 29946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:24.761519 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 29946
I20260928 16:34:24.761636 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:34:24.761935 28139 external_mini_cluster.cc:1428] Setting key 2be18e92d003f36f3b010e4d45a48677
I20260928 16:34:24.764005 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:24.749649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:45973 (local address 127.27.122.254:39421)
0928 16:34:24.749810 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:24.749816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:24.751060 (+ 1244us) server_negotiation.cc:407] Connection header received
0928 16:34:24.752230 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:24.752234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:24.752294 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:24.752376 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:24.754046 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.754946 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.756803 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.757113 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.757196 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:24.760125 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:24.760129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:24.760132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:24.760161 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:24.761851 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:24.762464 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:24.762469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:24.762470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:24.762561 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:24.762909 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:24.762914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:24.762915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:24.763112 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:24.763237 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:24.763705 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 16:34:24.763837 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:34:24.764705 29970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:24.750465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:39421 (local address 127.27.122.193:45973)
0928 16:34:24.750894 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:24.750943 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:24.752026 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:24.752590 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:24.752600 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:24.753124 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:24.753846 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:24.753870 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.755112 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.755119 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:24.756676 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:24.756687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.757281 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.757288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:24.757503 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:24.758113 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:24.758137 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:24.759944 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:24.761986 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:24.761991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:24.762004 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:24.762346 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:24.762661 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:24.762666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:24.762668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:24.762808 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:24.763224 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:24.763231 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:24.763467 (+ 236us) client_negotiation.cc:769] Sending connection context
0928 16:34:24.763661 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 16:34:24.763957 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":123,"threads_started":1}
I20260928 16:34:24.768316 30078 heartbeater.cc:347] Connected to a master server at 127.27.122.254:39421
I20260928 16:34:24.768488 30078 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:24.769030 30078 heartbeater.cc:511] Master 127.27.122.254:39421 requested a full tablet report, sending...
I20260928 16:34:24.770608 29873 ts_manager.cc:194] Registered new tserver with Master: 00e132429f6d4c5690730ec900474d9c (127.27.122.193:40857)
I20260928 16:34:24.771701 29873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:45973
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:34:24.776059 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:24.768526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:59495 (local address 127.27.122.254:39421)
0928 16:34:24.768646 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:24.768649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:24.769056 (+ 407us) server_negotiation.cc:407] Connection header received
0928 16:34:24.769218 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:24.769221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:24.769269 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:24.769354 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:24.769857 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.770384 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.771257 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.771481 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.771607 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:24.772772 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:24.772776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:24.772779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:24.772808 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:24.774535 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:24.774927 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:24.774930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:24.774931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:24.775002 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:24.775245 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:24.775248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:24.775249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:24.775434 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:24.775553 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:24.775813 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 16:34:24.775919 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:34:24.776081 30085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:24.768678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:39421 (local address 127.27.122.193:59495)
0928 16:34:24.768960 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:24.768974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:24.769103 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:24.769371 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:24.769374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:24.769562 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:24.769719 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:24.769730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.770517 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.770519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:24.771136 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:24.771145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.771604 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.771608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:24.771696 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:24.772208 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:24.772228 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:24.772623 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:24.774650 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:24.774652 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:24.774654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:24.774837 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:24.775092 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:24.775094 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:24.775095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:24.775167 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:24.775544 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:24.775547 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:24.775639 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 16:34:24.775757 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 16:34:24.775883 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:34:24.792346 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:39421
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:24.900543 30087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:24.900763 30087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:24.900805 30087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:24.904666 30087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:24.904716 30087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:24.904753 30087 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:34:24.904771 30087 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:34:24.904801 30087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:24.909845 30087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.122.254:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:24.910683 30087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:24.911592 30087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:24.920616 30093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:24.920569 30092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:24.920576 30095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:24.920876 30087 server_base.cc:1034] running on GCE node
I20260928 16:34:24.921363 30087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:24.921944 30087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:24.923131 30087 hybrid_clock.cc:648] HybridClock initialized: now 1790613264923121 us; error 30 us; skew 500 ppm
Sep 28 16:34:24 dist-test-slave-g9sh krb5kdc[29829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613264, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:24.925997 30087 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:34:24.926789 30087 webserver.cc:460] Webserver started at http://127.27.122.194:45409/ using document root <none> and password file <none>
I20260928 16:34:24.927376 30087 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:24.927462 30087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:24.927686 30087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:24.929638 30087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "dccb8b455e8e4425b6bd303f8b721403"
format_stamp: "Formatted at 2026-09-28 16:34:24 on dist-test-slave-g9sh"
server_key: "d9b2170553755f02a463154391e6bc4b"
server_key_iv: "460109f5b0202f5cf9404f12024e8774"
server_key_version: "encryptionkey@0"
I20260928 16:34:24.930181 30087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "dccb8b455e8e4425b6bd303f8b721403"
format_stamp: "Formatted at 2026-09-28 16:34:24 on dist-test-slave-g9sh"
server_key: "d9b2170553755f02a463154391e6bc4b"
server_key_iv: "460109f5b0202f5cf9404f12024e8774"
server_key_version: "encryptionkey@0"
I20260928 16:34:24.933650 30087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:34:24.935883 30102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:24.937089 30087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 16:34:24.937212 30087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "dccb8b455e8e4425b6bd303f8b721403"
format_stamp: "Formatted at 2026-09-28 16:34:24 on dist-test-slave-g9sh"
server_key: "d9b2170553755f02a463154391e6bc4b"
server_key_iv: "460109f5b0202f5cf9404f12024e8774"
server_key_version: "encryptionkey@0"
I20260928 16:34:24.937316 30087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:24.948602 30087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:24.951129 30087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:24.951320 30087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:24.952677 30087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:24.952745 30087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:24.952814 30087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:24.952852 30087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:34:24 dist-test-slave-g9sh krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613264, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:24.963850 30087 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:44817
I20260928 16:34:24.964545 30217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:44817 every 8 connection(s)
I20260928 16:34:24.964756 30087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:24.966600 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:24.953353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:33099 (local address 127.27.122.254:39421)
0928 16:34:24.953476 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:24.953487 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:24.954949 (+ 1462us) server_negotiation.cc:407] Connection header received
0928 16:34:24.956067 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:24.956070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:24.956132 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:24.956248 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:24.957901 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.958750 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.959945 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.960185 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.960296 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:24.962700 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:24.962704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:24.962707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:24.962739 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:24.964250 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:24.964944 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:24.964948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:24.964950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:24.965040 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:24.965440 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:24.965445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:24.965446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:24.965668 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:24.965736 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:24.966247 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 16:34:24.966395 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:34:24.967303 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:24.954359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:39421 (local address 127.27.122.194:33099)
0928 16:34:24.954803 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:24.954843 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:24.955862 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:24.956465 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:24.956475 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:24.956957 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:24.957687 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:24.957711 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.958874 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.958878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:24.959806 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:24.959819 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.960345 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.960350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:24.960526 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:24.961107 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:24.961123 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:24.962538 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:24.964370 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:24.964379 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:24.964395 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:24.964819 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:24.965141 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:24.965144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:24.965146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:24.965314 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:24.965805 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:24.965809 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:24.966010 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 16:34:24.966196 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 16:34:24.966510 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":121,"threads_started":1}
I20260928 16:34:24.967937 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 30087
I20260928 16:34:24.968065 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:34:24.968343 28139 external_mini_cluster.cc:1428] Setting key f3983d2f795f75288e493f69bbcc9661
I20260928 16:34:24.968973 30218 heartbeater.cc:347] Connected to a master server at 127.27.122.254:39421
I20260928 16:34:24.969170 30218 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:24.969702 30218 heartbeater.cc:511] Master 127.27.122.254:39421 requested a full tablet report, sending...
I20260928 16:34:24.970789 29873 ts_manager.cc:194] Registered new tserver with Master: dccb8b455e8e4425b6bd303f8b721403 (127.27.122.194:44817)
I20260928 16:34:24.971339 29873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:33099
I20260928 16:34:24.979022 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:24.970924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:48157 (local address 127.27.122.254:39421)
0928 16:34:24.971843 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:24.971847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:24.971861 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:34:24.971895 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:24.971898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:24.971937 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:24.972017 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:24.972426 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.973167 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.974264 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.974463 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.974538 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:24.975757 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:24.975760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:24.975762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:24.975783 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:24.977391 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:24.977888 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:24.977890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:24.977891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:24.977963 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:24.978237 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:24.978241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:24.978244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:24.978436 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:24.978515 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:24.978834 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 16:34:24.978921 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819}
I20260928 16:34:24.979144 30224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:24.970618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:39421 (local address 127.27.122.194:48157)
0928 16:34:24.970919 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:24.970936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:24.971026 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:24.972047 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:24.972049 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:24.972216 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:24.972309 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:24.972314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.973300 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.973307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:24.974162 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:24.974172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:24.974579 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:24.974585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:24.974712 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:24.975167 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:24.975187 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:24.975615 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:24.977497 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:24.977503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:24.977506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:24.977792 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:24.978033 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:24.978036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:24.978038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:24.978138 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:24.978543 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:24.978547 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:24.978657 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 16:34:24.978783 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 16:34:24.978927 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:34:24.999164 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:39421
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:25.107112 30227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:25.107335 30227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:25.107379 30227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:25.111248 30227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:25.111294 30227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:25.111331 30227 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 16:34:25.111346 30227 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:34:25.111385 30227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:25.116423 30227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.122.254:39421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:25.117451 30227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:25.118396 30227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:34:25.127949 30227 server_base.cc:1034] running on GCE node
W20260928 16:34:25.127976 30235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:25.128108 30232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:25.127975 30233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:25.128573 30227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:25.129257 30227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:25.130445 30227 hybrid_clock.cc:648] HybridClock initialized: now 1790613265130430 us; error 26 us; skew 500 ppm
Sep 28 16:34:25 dist-test-slave-g9sh krb5kdc[29829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613265, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:25.133172 30227 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:34:25.134099 30227 webserver.cc:460] Webserver started at http://127.27.122.195:41475/ using document root <none> and password file <none>
I20260928 16:34:25.134639 30227 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:25.134727 30227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:25.134956 30227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:25.137190 30227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "c87c9eb089bd4074b9864cba03cda1bd"
format_stamp: "Formatted at 2026-09-28 16:34:25 on dist-test-slave-g9sh"
server_key: "852af28775a0b435b516d25ff9e6aad9"
server_key_iv: "29999216885b6e04c3b316ca3ebfe9da"
server_key_version: "encryptionkey@0"
I20260928 16:34:25.137737 30227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "c87c9eb089bd4074b9864cba03cda1bd"
format_stamp: "Formatted at 2026-09-28 16:34:25 on dist-test-slave-g9sh"
server_key: "852af28775a0b435b516d25ff9e6aad9"
server_key_iv: "29999216885b6e04c3b316ca3ebfe9da"
server_key_version: "encryptionkey@0"
I20260928 16:34:25.141412 30227 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:34:25.143653 30242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:25.144865 30227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:25.145038 30227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "c87c9eb089bd4074b9864cba03cda1bd"
format_stamp: "Formatted at 2026-09-28 16:34:25 on dist-test-slave-g9sh"
server_key: "852af28775a0b435b516d25ff9e6aad9"
server_key_iv: "29999216885b6e04c3b316ca3ebfe9da"
server_key_version: "encryptionkey@0"
I20260928 16:34:25.145157 30227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:25.167632 30227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:25.171098 30227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:25.171300 30227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:25.172755 30227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:25.172827 30227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:25.172933 30227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:25.172976 30227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 28 16:34:25 dist-test-slave-g9sh krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613265, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:25.183768 30227 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:46879
I20260928 16:34:25.184047 30357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:46879 every 8 connection(s)
I20260928 16:34:25.184906 30227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:34:25.187095 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.173463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:53935 (local address 127.27.122.254:39421)
0928 16:34:25.173590 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.173594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.175229 (+ 1635us) server_negotiation.cc:407] Connection header received
0928 16:34:25.176368 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.176372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.176437 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.176532 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:25.178219 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.178776 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.180106 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.180340 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.180459 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.183357 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:25.183361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:25.183364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:25.183398 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:25.185141 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:25.185739 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:25.185743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:25.185744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:25.185843 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:25.186196 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:25.186198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:25.186199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:25.186367 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:25.186443 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.186867 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.186967 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:34:25.187855 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.174557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:39421 (local address 127.27.122.195:53935)
0928 16:34:25.175079 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.175124 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.176149 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.176741 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.176750 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.177272 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:25.178026 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.178049 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.178914 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.178917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.179975 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.179985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.180518 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.180523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.180692 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.181764 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:25.181780 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:25.183189 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:25.185247 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:25.185251 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:25.185262 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:25.185610 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:25.185941 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:25.185945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:25.185947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:25.186110 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:25.186457 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:25.186462 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:25.186671 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.186818 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.187046 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
I20260928 16:34:25.189036 30358 heartbeater.cc:347] Connected to a master server at 127.27.122.254:39421
I20260928 16:34:25.189181 30358 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:25.189700 30358 heartbeater.cc:511] Master 127.27.122.254:39421 requested a full tablet report, sending...
I20260928 16:34:25.190838 29874 ts_manager.cc:194] Registered new tserver with Master: c87c9eb089bd4074b9864cba03cda1bd (127.27.122.195:46879)
I20260928 16:34:25.191612 29874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:53935
I20260928 16:34:25.194860 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 30227
I20260928 16:34:25.194963 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:34:25.195317 28139 external_mini_cluster.cc:1428] Setting key af00d8ad5f8a9e1f9f3cf875d3cc80f3
I20260928 16:34:25.197607 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:34:25.198436 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.191301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:52305 (local address 127.27.122.254:39421)
0928 16:34:25.191441 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.191445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.191590 (+ 145us) server_negotiation.cc:407] Connection header received
0928 16:34:25.191694 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.191697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.191742 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.191809 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:25.192221 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.192691 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.193765 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.193975 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.194056 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.195204 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:25.195207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:25.195209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:25.195229 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:25.196749 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:25.197207 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:25.197210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:25.197212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:25.197295 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:25.197567 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:25.197572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:25.197573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:25.197801 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:25.197905 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.198180 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.198287 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:34:25.198446 30364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.191222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:39421 (local address 127.27.122.195:52305)
0928 16:34:25.191491 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.191504 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.191620 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.191824 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.191827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.192010 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:25.192107 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.192112 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.192823 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.192826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.193647 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.193657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.194069 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.194071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.194142 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.194626 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:25.194646 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:25.195073 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:25.196870 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:25.196873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:25.196875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:25.197100 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:25.197386 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:25.197390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:25.197392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:25.197475 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:25.197897 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:25.197900 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:25.197996 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.198120 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.198258 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
Sep 28 16:34:25 dist-test-slave-g9sh krb5kdc[29829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:34:25 dist-test-slave-g9sh krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:25.222847 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.214747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34018 (local address 127.27.122.254:39421)
0928 16:34:25.214872 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.214876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.214888 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:34:25.214923 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.214925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.214961 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.215021 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:25.215732 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.216254 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.217174 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.217394 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.217506 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.219862 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:25.219872 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:25.219876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:25.219897 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:25.221193 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:25.221623 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:25.221626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:25.221630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:25.221709 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:25.221988 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:25.221992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:25.221993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:25.222176 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:25.222318 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.222559 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.222675 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:34:25.225924 29874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:34:25.229010 29874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:25.243449 30376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.237233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40857 (local address 127.0.0.1:55220)
0928 16:34:25.238327 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.238337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.238410 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.238903 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.238909 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.238932 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:25.239071 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.239077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.240479 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.240486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.241794 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.241804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.242889 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.242893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.243002 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.243039 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.243165 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.243285 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":19,"thread_start_us":110,"threads_started":1}
I20260928 16:34:25.243773 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.237926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.27.122.195:46879)
0928 16:34:25.238385 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.238396 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.238636 (+ 240us) server_negotiation.cc:407] Connection header received
0928 16:34:25.238734 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.238739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.238915 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.239062 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:25.239523 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.240636 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.242247 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.242786 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.242950 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.243077 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.243522 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.243630 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":106,"threads_started":1}
I20260928 16:34:25.243857 30375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.237518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:46879 (local address 127.0.0.1:48420)
0928 16:34:25.238515 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.238528 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.238630 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.239221 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.239224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.239267 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:25.239390 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.239397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.240749 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.240755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.242126 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.242142 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.243143 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.243149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.243267 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.243287 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.243355 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.243409 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":53,"spinlock_wait_cycles":39168,"thread_start_us":85,"threads_started":1}
I20260928 16:34:25.243466 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.237924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.27.122.193:40857)
0928 16:34:25.238307 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.238323 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.238400 (+ 77us) server_negotiation.cc:407] Connection header received
0928 16:34:25.238507 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.238513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.238750 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.238918 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:25.239203 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.240360 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.241913 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.242754 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.242897 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.243025 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.243212 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.243321 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":75,"threads_started":1}
I20260928 16:34:25.245467 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.237122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:44817 (local address 127.0.0.1:52256)
0928 16:34:25.237948 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.237996 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.238112 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.239221 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.239224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.239254 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:25.239391 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.239397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.242111 (+ 2714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.242114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.243234 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.243244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.245071 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.245075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.245173 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.245196 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.245278 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.245347 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":35,"thread_start_us":84,"threads_started":1}
I20260928 16:34:25.245555 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.237257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.27.122.194:44817)
0928 16:34:25.238662 (+ 1405us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.238675 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.238695 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:34:25.238760 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.238765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.238915 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.239072 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:25.240797 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.241991 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.243370 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.244721 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.244856 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.244997 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.245328 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.245421 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1277,"thread_start_us":86,"threads_started":1}
I20260928 16:34:25.246879 30011 tablet_service.cc:1467] Processing CreateTablet for tablet e0fd994e96a14020bca33cc803510019 (DEFAULT_TABLE table=test-table [id=c12bd8f669cb4a18941fe97e194a6a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:25.246893 30292 tablet_service.cc:1467] Processing CreateTablet for tablet e0fd994e96a14020bca33cc803510019 (DEFAULT_TABLE table=test-table [id=c12bd8f669cb4a18941fe97e194a6a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:25.247934 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fd994e96a14020bca33cc803510019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:25.247952 30292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fd994e96a14020bca33cc803510019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:25.248303 30152 tablet_service.cc:1467] Processing CreateTablet for tablet e0fd994e96a14020bca33cc803510019 (DEFAULT_TABLE table=test-table [id=c12bd8f669cb4a18941fe97e194a6a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:25.249243 30152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fd994e96a14020bca33cc803510019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:25.254510 30381 tablet_bootstrap.cc:492] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c: Bootstrap starting.
I20260928 16:34:25.256381 30383 tablet_bootstrap.cc:492] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403: Bootstrap starting.
I20260928 16:34:25.256697 30381 tablet_bootstrap.cc:654] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:25.257007 30382 tablet_bootstrap.cc:492] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd: Bootstrap starting.
I20260928 16:34:25.257618 30381 log.cc:824] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:25.258481 30383 tablet_bootstrap.cc:654] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:25.258896 30382 tablet_bootstrap.cc:654] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:25.259294 30383 log.cc:824] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:25.259617 30382 log.cc:824] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:25.259860 30381 tablet_bootstrap.cc:492] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c: No bootstrap required, opened a new log
I20260928 16:34:25.260073 30381 ts_tablet_manager.cc:1397] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:34:25.261212 30383 tablet_bootstrap.cc:492] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403: No bootstrap required, opened a new log
I20260928 16:34:25.261397 30383 ts_tablet_manager.cc:1397] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:34:25.263924 30381 raft_consensus.cc:348] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.264217 30381 raft_consensus.cc:374] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:25.264302 30381 raft_consensus.cc:729] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e132429f6d4c5690730ec900474d9c, State: Initialized, Role: FOLLOWER
I20260928 16:34:25.265156 30381 consensus_queue.cc:260] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.265188 30383 raft_consensus.cc:348] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.265395 30383 raft_consensus.cc:374] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:25.265452 30383 raft_consensus.cc:729] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccb8b455e8e4425b6bd303f8b721403, State: Initialized, Role: FOLLOWER
I20260928 16:34:25.265904 30383 consensus_queue.cc:260] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.266150 30382 tablet_bootstrap.cc:492] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd: No bootstrap required, opened a new log
I20260928 16:34:25.266247 30381 ts_tablet_manager.cc:1428] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 16:34:25.266351 30382 ts_tablet_manager.cc:1397] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260928 16:34:25.266604 30078 heartbeater.cc:503] Master 127.27.122.254:39421 was elected leader, sending a full tablet report...
I20260928 16:34:25.266860 30218 heartbeater.cc:503] Master 127.27.122.254:39421 was elected leader, sending a full tablet report...
I20260928 16:34:25.266861 30383 ts_tablet_manager.cc:1428] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 16:34:25.270180 30382 raft_consensus.cc:348] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.270443 30382 raft_consensus.cc:374] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:25.270520 30382 raft_consensus.cc:729] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87c9eb089bd4074b9864cba03cda1bd, State: Initialized, Role: FOLLOWER
I20260928 16:34:25.270881 30382 consensus_queue.cc:260] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.271476 30358 heartbeater.cc:503] Master 127.27.122.254:39421 was elected leader, sending a full tablet report...
I20260928 16:34:25.272591 30382 ts_tablet_manager.cc:1428] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:34:25.275903 30387 raft_consensus.cc:482] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:25.276048 30387 raft_consensus.cc:504] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.277100 30387 leader_election.cc:288] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dccb8b455e8e4425b6bd303f8b721403 (127.27.122.194:44817), c87c9eb089bd4074b9864cba03cda1bd (127.27.122.195:46879)
I20260928 16:34:25.280845 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.277404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:57415 (local address 127.27.122.194:44817)
0928 16:34:25.277517 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.277521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.277774 (+ 253us) server_negotiation.cc:407] Connection header received
0928 16:34:25.277931 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.277937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.277987 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.278043 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:25.278336 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.278857 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.279927 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.280303 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.280431 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.280480 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.280672 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.280735 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:34:25.280861 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.277284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:44817 (local address 127.27.122.193:57415)
0928 16:34:25.277674 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.277689 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.277816 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.278081 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.278085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.278107 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:25.278216 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.278222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.278991 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.278998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.279812 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.279821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.280427 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.280431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.280529 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.280552 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.280656 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.280727 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":57,"threads_started":1}
I20260928 16:34:25.281515 30172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd994e96a14020bca33cc803510019" candidate_uuid: "00e132429f6d4c5690730ec900474d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccb8b455e8e4425b6bd303f8b721403" is_pre_election: true
I20260928 16:34:25.281781 30172 raft_consensus.cc:2454] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00e132429f6d4c5690730ec900474d9c in term 0.
I20260928 16:34:25.281914 30390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.277284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:46879 (local address 127.27.122.193:54689)
0928 16:34:25.277728 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.277739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.277853 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.278126 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.278130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.278150 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:25.278248 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.278253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.279924 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.279926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.280595 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.280602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.281472 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.281475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.281576 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.281614 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.281702 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.281772 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":106,"thread_start_us":102,"threads_started":1}
I20260928 16:34:25.281945 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.277440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:54689 (local address 127.27.122.195:46879)
0928 16:34:25.277541 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.277545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.277816 (+ 271us) server_negotiation.cc:407] Connection header received
0928 16:34:25.277946 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.277949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.277994 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.278103 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:25.278379 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.278857 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.280714 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.281363 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.281471 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.281516 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.281754 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.281840 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:34:25.282238 29964 leader_election.cc:302] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e132429f6d4c5690730ec900474d9c, dccb8b455e8e4425b6bd303f8b721403; no voters:
I20260928 16:34:25.282497 30387 raft_consensus.cc:2781] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:25.282582 30387 raft_consensus.cc:482] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:25.282613 30387 raft_consensus.cc:3037] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:25.282689 30312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd994e96a14020bca33cc803510019" candidate_uuid: "00e132429f6d4c5690730ec900474d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87c9eb089bd4074b9864cba03cda1bd" is_pre_election: true
I20260928 16:34:25.282935 30312 raft_consensus.cc:2454] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00e132429f6d4c5690730ec900474d9c in term 0.
I20260928 16:34:25.283931 30387 raft_consensus.cc:504] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.284262 30387 leader_election.cc:288] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [CANDIDATE]: Term 1 election: Requested vote from peers dccb8b455e8e4425b6bd303f8b721403 (127.27.122.194:44817), c87c9eb089bd4074b9864cba03cda1bd (127.27.122.195:46879)
I20260928 16:34:25.284617 30172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd994e96a14020bca33cc803510019" candidate_uuid: "00e132429f6d4c5690730ec900474d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccb8b455e8e4425b6bd303f8b721403"
I20260928 16:34:25.284736 30312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fd994e96a14020bca33cc803510019" candidate_uuid: "00e132429f6d4c5690730ec900474d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87c9eb089bd4074b9864cba03cda1bd"
I20260928 16:34:25.284791 30172 raft_consensus.cc:3037] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:25.284848 30312 raft_consensus.cc:3037] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:25.286080 30172 raft_consensus.cc:2454] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00e132429f6d4c5690730ec900474d9c in term 1.
I20260928 16:34:25.286080 30312 raft_consensus.cc:2454] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00e132429f6d4c5690730ec900474d9c in term 1.
I20260928 16:34:25.286360 29964 leader_election.cc:302] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e132429f6d4c5690730ec900474d9c, dccb8b455e8e4425b6bd303f8b721403; no voters:
I20260928 16:34:25.286605 30387 raft_consensus.cc:2781] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:25.286759 30387 raft_consensus.cc:686] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [term 1 LEADER]: Becoming Leader. State: Replica: 00e132429f6d4c5690730ec900474d9c, State: Running, Role: LEADER
I20260928 16:34:25.287051 30387 consensus_queue.cc:237] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.290174 29873 catalog_manager.cc:5495] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 00e132429f6d4c5690730ec900474d9c (127.27.122.193). New cstate: current_term: 1 leader_uuid: "00e132429f6d4c5690730ec900474d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } health_report { overall_health: UNKNOWN } } }
Sep 28 16:34:25 dist-test-slave-g9sh krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613264, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:25.327222 30399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.318511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:39421 (local address 127.0.0.1:34032)
0928 16:34:25.318774 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.318786 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.318879 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.319110 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.319112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.319469 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:25.319560 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.319566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.320250 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.320253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.320785 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.320791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.321248 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.321250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.321312 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.321918 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:25.321930 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:25.323276 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:25.325081 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:25.325093 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:25.325102 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:25.325441 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:25.325794 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:25.325799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:25.325803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:25.325905 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:25.326434 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:25.326442 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:25.326590 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.326728 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.326922 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260928 16:34:25.327237 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.318598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.27.122.254:39421)
0928 16:34:25.318759 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.318766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.318875 (+ 109us) server_negotiation.cc:407] Connection header received
0928 16:34:25.318944 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.318946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.318986 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.319054 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:25.319660 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.320161 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.320876 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.321146 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.321230 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.323398 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:25.323401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:25.323403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:25.323424 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:25.324964 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:25.325574 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:25.325579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:25.325581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:25.325687 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:25.326041 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:25.326044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:25.326046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:25.326297 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:25.326432 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.326810 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.327004 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":82}
I20260928 16:34:25.333439 29873 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:34032:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:34:25.334371 29873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:25.339395 30011 tablet_service.cc:1467] Processing CreateTablet for tablet 869ff78b17974dada4222eafbe64f966 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7a2fe18cbc3644b0bd934baae7013308]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:34:25.339641 30152 tablet_service.cc:1467] Processing CreateTablet for tablet 869ff78b17974dada4222eafbe64f966 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7a2fe18cbc3644b0bd934baae7013308]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:34:25.340189 30011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 869ff78b17974dada4222eafbe64f966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:25.340189 30152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 869ff78b17974dada4222eafbe64f966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:25.343201 30381 tablet_bootstrap.cc:492] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: Bootstrap starting.
I20260928 16:34:25.343201 30383 tablet_bootstrap.cc:492] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403: Bootstrap starting.
I20260928 16:34:25.343667 30292 tablet_service.cc:1467] Processing CreateTablet for tablet 869ff78b17974dada4222eafbe64f966 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7a2fe18cbc3644b0bd934baae7013308]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:34:25.344221 30292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 869ff78b17974dada4222eafbe64f966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:25.344219 30383 tablet_bootstrap.cc:654] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:25.344219 30381 tablet_bootstrap.cc:654] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:25.345821 30381 tablet_bootstrap.cc:492] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: No bootstrap required, opened a new log
I20260928 16:34:25.345860 30383 tablet_bootstrap.cc:492] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403: No bootstrap required, opened a new log
I20260928 16:34:25.345933 30383 ts_tablet_manager.cc:1397] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:34:25.345932 30381 ts_tablet_manager.cc:1397] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 16:34:25.346350 30381 raft_consensus.cc:348] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.346469 30381 raft_consensus.cc:374] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:25.346405 30383 raft_consensus.cc:348] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.346510 30381 raft_consensus.cc:729] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e132429f6d4c5690730ec900474d9c, State: Initialized, Role: FOLLOWER
I20260928 16:34:25.346520 30383 raft_consensus.cc:374] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:25.346552 30383 raft_consensus.cc:729] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccb8b455e8e4425b6bd303f8b721403, State: Initialized, Role: FOLLOWER
I20260928 16:34:25.346678 30381 consensus_queue.cc:260] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.346681 30383 consensus_queue.cc:260] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.346939 30383 ts_tablet_manager.cc:1428] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:25.346980 30381 ts_tablet_manager.cc:1428] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 16:34:25.347151 30387 tablet_replica.cc:439] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } } }
I20260928 16:34:25.347281 30382 tablet_bootstrap.cc:492] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd: Bootstrap starting.
I20260928 16:34:25.347280 30387 tablet_replica.cc:442] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:25.348284 30382 tablet_bootstrap.cc:654] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:25.346946 30388 tablet_replica.cc:439] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } } }
I20260928 16:34:25.348569 30388 tablet_replica.cc:442] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:25.349857 30382 tablet_bootstrap.cc:492] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd: No bootstrap required, opened a new log
I20260928 16:34:25.349970 30382 ts_tablet_manager.cc:1397] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:34:25.350435 30382 raft_consensus.cc:348] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.350551 30382 raft_consensus.cc:374] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:25.350593 30382 raft_consensus.cc:729] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87c9eb089bd4074b9864cba03cda1bd, State: Initialized, Role: FOLLOWER
I20260928 16:34:25.350721 30382 consensus_queue.cc:260] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.350972 30382 ts_tablet_manager.cc:1428] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:34:25.351008 30389 tablet_replica.cc:439] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } } }
I20260928 16:34:25.351155 30389 tablet_replica.cc:442] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 16:34:25.435959 30359 tablet.cc:2369] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:25.461043 30387 raft_consensus.cc:482] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:25.461174 30387 raft_consensus.cc:504] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.461616 30387 leader_election.cc:288] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dccb8b455e8e4425b6bd303f8b721403 (127.27.122.194:44817), c87c9eb089bd4074b9864cba03cda1bd (127.27.122.195:46879)
I20260928 16:34:25.462026 30172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869ff78b17974dada4222eafbe64f966" candidate_uuid: "00e132429f6d4c5690730ec900474d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccb8b455e8e4425b6bd303f8b721403" is_pre_election: true
I20260928 16:34:25.462116 30312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869ff78b17974dada4222eafbe64f966" candidate_uuid: "00e132429f6d4c5690730ec900474d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87c9eb089bd4074b9864cba03cda1bd" is_pre_election: true
I20260928 16:34:25.462188 30172 raft_consensus.cc:2454] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00e132429f6d4c5690730ec900474d9c in term 0.
I20260928 16:34:25.462244 30312 raft_consensus.cc:2454] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00e132429f6d4c5690730ec900474d9c in term 0.
I20260928 16:34:25.462508 29964 leader_election.cc:302] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00e132429f6d4c5690730ec900474d9c, c87c9eb089bd4074b9864cba03cda1bd, dccb8b455e8e4425b6bd303f8b721403; no voters:
I20260928 16:34:25.462743 30387 raft_consensus.cc:2781] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:25.462819 30387 raft_consensus.cc:482] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:25.462872 30387 raft_consensus.cc:3037] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:25.463876 30387 raft_consensus.cc:504] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.464224 30387 leader_election.cc:288] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [CANDIDATE]: Term 1 election: Requested vote from peers dccb8b455e8e4425b6bd303f8b721403 (127.27.122.194:44817), c87c9eb089bd4074b9864cba03cda1bd (127.27.122.195:46879)
I20260928 16:34:25.464632 30172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869ff78b17974dada4222eafbe64f966" candidate_uuid: "00e132429f6d4c5690730ec900474d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccb8b455e8e4425b6bd303f8b721403"
I20260928 16:34:25.464653 30312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "869ff78b17974dada4222eafbe64f966" candidate_uuid: "00e132429f6d4c5690730ec900474d9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87c9eb089bd4074b9864cba03cda1bd"
I20260928 16:34:25.464761 30312 raft_consensus.cc:3037] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:25.464761 30172 raft_consensus.cc:3037] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:25.465903 30312 raft_consensus.cc:2454] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00e132429f6d4c5690730ec900474d9c in term 1.
I20260928 16:34:25.465919 30172 raft_consensus.cc:2454] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00e132429f6d4c5690730ec900474d9c in term 1.
W20260928 16:34:25.466044 30219 tablet.cc:2369] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:25.466204 29963 leader_election.cc:302] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e132429f6d4c5690730ec900474d9c, c87c9eb089bd4074b9864cba03cda1bd; no voters:
I20260928 16:34:25.466423 30387 raft_consensus.cc:2781] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:25.466542 30387 raft_consensus.cc:686] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [term 1 LEADER]: Becoming Leader. State: Replica: 00e132429f6d4c5690730ec900474d9c, State: Running, Role: LEADER
I20260928 16:34:25.466694 30387 consensus_queue.cc:237] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.467419 30392 tablet_replica.cc:439] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: TxnStatusTablet state changed. Reason: New leader 00e132429f6d4c5690730ec900474d9c. Latest consensus state: current_term: 1 leader_uuid: "00e132429f6d4c5690730ec900474d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } } }
I20260928 16:34:25.467530 30392 tablet_replica.cc:442] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:34:25.468119 30404 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:34:25.468382 30404 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:34:25.468456 29873 catalog_manager.cc:5495] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 00e132429f6d4c5690730ec900474d9c (127.27.122.193). New cstate: current_term: 1 leader_uuid: "00e132429f6d4c5690730ec900474d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } health_report { overall_health: UNKNOWN } } }
I20260928 16:34:25.509629 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.505042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.27.122.193:40857)
0928 16:34:25.505203 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.505208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.505221 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:34:25.505302 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.505305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.505369 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.505469 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:25.505776 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.506578 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.507857 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.508224 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.508329 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.508668 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:25.509209 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:25.509316 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.509432 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.509495 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:34:25.509629 30399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.504513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40857 (local address 127.0.0.1:55234)
0928 16:34:25.504755 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.504771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.504870 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.505509 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.505512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.505522 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:34:25.505656 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.505662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.506741 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.506745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.507727 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.507737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.508343 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.508346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.508472 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.508521 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:34:25.509321 (+ 800us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:34:25.509335 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.509400 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.509494 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
W20260928 16:34:25.512455 30079 tablet.cc:2369] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:25.517617 30172 raft_consensus.cc:1261] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Refusing update from remote peer 00e132429f6d4c5690730ec900474d9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:25.517618 30312 raft_consensus.cc:1261] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Refusing update from remote peer 00e132429f6d4c5690730ec900474d9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:25.518314 30392 consensus_queue.cc:1035] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:25.518517 30387 consensus_queue.cc:1035] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:25.521958 30388 tablet_replica.cc:439] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403: TxnStatusTablet state changed. Reason: New leader 00e132429f6d4c5690730ec900474d9c. Latest consensus state: current_term: 1 leader_uuid: "00e132429f6d4c5690730ec900474d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } } }
I20260928 16:34:25.522084 30388 tablet_replica.cc:442] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:25.522367 30389 tablet_replica.cc:439] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd: TxnStatusTablet state changed. Reason: New leader 00e132429f6d4c5690730ec900474d9c. Latest consensus state: current_term: 1 leader_uuid: "00e132429f6d4c5690730ec900474d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } } }
I20260928 16:34:25.522488 30389 tablet_replica.cc:442] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:25.525525 30392 tablet_replica.cc:439] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00e132429f6d4c5690730ec900474d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } } }
I20260928 16:34:25.525646 30407 mvcc.cc:205] Tried to move back new op lower bound from 7334351935553196032 to 7334351935353122816. Current Snapshot: MvccSnapshot[applied={T|T < 7334351935553196032}]
I20260928 16:34:25.525650 30392 tablet_replica.cc:442] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:34:25.528280 30388 tablet_replica.cc:439] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00e132429f6d4c5690730ec900474d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } } }
I20260928 16:34:25.528750 30388 tablet_replica.cc:442] T 869ff78b17974dada4222eafbe64f966 P dccb8b455e8e4425b6bd303f8b721403: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:25.529191 30408 mvcc.cc:205] Tried to move back new op lower bound from 7334351935553196032 to 7334351935353122816. Current Snapshot: MvccSnapshot[applied={T|T < 7334351935553196032}]
I20260928 16:34:25.531417 30392 tablet_replica.cc:439] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "00e132429f6d4c5690730ec900474d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } } }
I20260928 16:34:25.531518 30392 tablet_replica.cc:442] T 869ff78b17974dada4222eafbe64f966 P 00e132429f6d4c5690730ec900474d9c: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:34:25.532528 30389 tablet_replica.cc:439] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "00e132429f6d4c5690730ec900474d9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } } }
I20260928 16:34:25.532639 30389 tablet_replica.cc:442] T 869ff78b17974dada4222eafbe64f966 P c87c9eb089bd4074b9864cba03cda1bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:25.538015 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.532971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.27.122.193:40857)
0928 16:34:25.533085 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.533089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.533102 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:34:25.533143 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.533146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.533212 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.533317 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:25.533587 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.534324 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.535499 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.535839 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.535960 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.537274 (+ 1314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:25.537356 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:25.537478 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.537840 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.537893 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:34:25.548873 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.545298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:38235 (local address 127.27.122.193:40857)
0928 16:34:25.545395 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.545399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.545594 (+ 195us) server_negotiation.cc:407] Connection header received
0928 16:34:25.545704 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.545708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.545754 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.545831 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:25.546053 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.546769 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.547594 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.547933 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.548031 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.548375 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:25.548447 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:25.548533 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.548651 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.548696 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":39}
I20260928 16:34:25.548962 30419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.545245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40857 (local address 127.27.122.193:38235)
0928 16:34:25.545505 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.545522 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.545620 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.545845 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.545847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.545852 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:34:25.545956 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.545960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.546885 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.546888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.547488 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.547495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.548029 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.548032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.548197 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.548237 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:34:25.548553 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:34:25.548559 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.548616 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.548713 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":29,"thread_start_us":98,"threads_started":1}
I20260928 16:34:25.555662 30172 raft_consensus.cc:1261] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Refusing update from remote peer 00e132429f6d4c5690730ec900474d9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:25.555652 30312 raft_consensus.cc:1261] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Refusing update from remote peer 00e132429f6d4c5690730ec900474d9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:25.556062 30414 consensus_queue.cc:1035] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:25.556289 30387 consensus_queue.cc:1035] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:25.561277 30409 mvcc.cc:205] Tried to move back new op lower bound from 7334351935712956416 to 7334351934618353664. Current Snapshot: MvccSnapshot[applied={T|T < 7334351935712956416}]
I20260928 16:34:25.593008 29873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:25.593194 29873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:25.595443 29873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6: Sending DeleteTablet for 3 replicas of tablet e0fd994e96a14020bca33cc803510019
I20260928 16:34:25.596156 30152 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fd994e96a14020bca33cc803510019 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:25 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:52256
I20260928 16:34:25.596297 30292 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fd994e96a14020bca33cc803510019 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:25 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:48420
I20260928 16:34:25.596292 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet e0fd994e96a14020bca33cc803510019 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:25 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:55220
I20260928 16:34:25.596673 30425 tablet_replica.cc:330] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd: stopping tablet replica
I20260928 16:34:25.597000 30425 raft_consensus.cc:2229] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:25.597047 30424 tablet_replica.cc:330] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403: stopping tablet replica
I20260928 16:34:25.597209 30425 raft_consensus.cc:2258] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:25.597306 30424 raft_consensus.cc:2229] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:25.597309 29873 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 16:34:25.597543 30426 tablet_replica.cc:330] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c: stopping tablet replica
W20260928 16:34:25.597761 29873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:25.597788 30426 raft_consensus.cc:2229] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:25.598107 30426 raft_consensus.cc:2258] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:25.598807 30425 ts_tablet_manager.cc:1905] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:25.599404 30426 ts_tablet_manager.cc:1905] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:25.602356 30426 ts_tablet_manager.cc:1918] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:25.602357 30425 ts_tablet_manager.cc:1918] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:25.602453 30425 log.cc:1196] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/e0fd994e96a14020bca33cc803510019
I20260928 16:34:25.602463 30426 log.cc:1196] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/wal/wals/e0fd994e96a14020bca33cc803510019
I20260928 16:34:25.603030 30010 tablet_service.cc:1467] Processing CreateTablet for tablet 0d601ec9bbae4810b9bd04c1d1bb391c (DEFAULT_TABLE table=test-table [id=0afbfd8be52043bab1890c98f457798b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:25.603267 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d601ec9bbae4810b9bd04c1d1bb391c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:25.603221 30292 tablet_service.cc:1467] Processing CreateTablet for tablet 0d601ec9bbae4810b9bd04c1d1bb391c (DEFAULT_TABLE table=test-table [id=0afbfd8be52043bab1890c98f457798b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:25.603297 30152 tablet_service.cc:1467] Processing CreateTablet for tablet 0d601ec9bbae4810b9bd04c1d1bb391c (DEFAULT_TABLE table=test-table [id=0afbfd8be52043bab1890c98f457798b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:25.603444 30292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d601ec9bbae4810b9bd04c1d1bb391c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:25.603554 30152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d601ec9bbae4810b9bd04c1d1bb391c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:25.605496 30381 tablet_bootstrap.cc:492] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c: Bootstrap starting.
I20260928 16:34:25.605566 30382 tablet_bootstrap.cc:492] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd: Bootstrap starting.
I20260928 16:34:25.606426 30382 tablet_bootstrap.cc:654] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:25.606438 30381 tablet_bootstrap.cc:654] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:25.607050 30383 tablet_bootstrap.cc:492] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403: Bootstrap starting.
I20260928 16:34:25.602782 30425 ts_tablet_manager.cc:1939] T e0fd994e96a14020bca33cc803510019 P c87c9eb089bd4074b9864cba03cda1bd: Deleting consensus metadata
I20260928 16:34:25.607986 30383 tablet_bootstrap.cc:654] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:25.608057 30382 tablet_bootstrap.cc:492] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd: No bootstrap required, opened a new log
I20260928 16:34:25.608126 30382 ts_tablet_manager.cc:1397] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 16:34:25.609081 30382 raft_consensus.cc:348] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.609198 30382 raft_consensus.cc:374] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:25.609237 30382 raft_consensus.cc:729] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87c9eb089bd4074b9864cba03cda1bd, State: Initialized, Role: FOLLOWER
I20260928 16:34:25.609275 30381 tablet_bootstrap.cc:492] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c: No bootstrap required, opened a new log
I20260928 16:34:25.609342 30381 ts_tablet_manager.cc:1397] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260928 16:34:25.609361 30382 consensus_queue.cc:260] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.609674 30382 ts_tablet_manager.cc:1428] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 16:34:25.609795 30381 raft_consensus.cc:348] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.609889 30381 raft_consensus.cc:374] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:25.609918 30381 raft_consensus.cc:729] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e132429f6d4c5690730ec900474d9c, State: Initialized, Role: FOLLOWER
I20260928 16:34:25.610028 30381 consensus_queue.cc:260] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.610253 30381 ts_tablet_manager.cc:1428] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 16:34:25.610528 29859 catalog_manager.cc:4841] TS c87c9eb089bd4074b9864cba03cda1bd (127.27.122.195:46879): tablet e0fd994e96a14020bca33cc803510019 (table test-table [id=c12bd8f669cb4a18941fe97e194a6a8c]) successfully deleted
I20260928 16:34:25.610782 30424 raft_consensus.cc:2258] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:25.612591 30424 ts_tablet_manager.cc:1905] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:25.613857 30426 ts_tablet_manager.cc:1939] T e0fd994e96a14020bca33cc803510019 P 00e132429f6d4c5690730ec900474d9c: Deleting consensus metadata
I20260928 16:34:25.614949 29858 catalog_manager.cc:4841] TS 00e132429f6d4c5690730ec900474d9c (127.27.122.193:40857): tablet e0fd994e96a14020bca33cc803510019 (table test-table [id=c12bd8f669cb4a18941fe97e194a6a8c]) successfully deleted
I20260928 16:34:25.615672 30424 ts_tablet_manager.cc:1918] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:25.615784 30424 log.cc:1196] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/e0fd994e96a14020bca33cc803510019
I20260928 16:34:25.615832 30383 tablet_bootstrap.cc:492] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403: No bootstrap required, opened a new log
I20260928 16:34:25.615922 30383 ts_tablet_manager.cc:1397] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 16:34:25.616135 30424 ts_tablet_manager.cc:1939] T e0fd994e96a14020bca33cc803510019 P dccb8b455e8e4425b6bd303f8b721403: Deleting consensus metadata
I20260928 16:34:25.616394 30383 raft_consensus.cc:348] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.616516 30383 raft_consensus.cc:374] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:25.616551 30383 raft_consensus.cc:729] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccb8b455e8e4425b6bd303f8b721403, State: Initialized, Role: FOLLOWER
I20260928 16:34:25.616681 30383 consensus_queue.cc:260] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.616984 30383 ts_tablet_manager.cc:1428] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:25.617420 29860 catalog_manager.cc:4841] TS dccb8b455e8e4425b6bd303f8b721403 (127.27.122.194:44817): tablet e0fd994e96a14020bca33cc803510019 (table test-table [id=c12bd8f669cb4a18941fe97e194a6a8c]) successfully deleted
I20260928 16:34:25.635670 30389 raft_consensus.cc:482] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:25.635820 30389 raft_consensus.cc:504] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.637035 30389 leader_election.cc:288] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00e132429f6d4c5690730ec900474d9c (127.27.122.193:40857), dccb8b455e8e4425b6bd303f8b721403 (127.27.122.194:44817)
I20260928 16:34:25.641551 30377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.637392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:43423 (local address 127.27.122.194:44817)
0928 16:34:25.637529 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.637534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.637684 (+ 150us) server_negotiation.cc:407] Connection header received
0928 16:34:25.637795 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.637798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.637844 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.637916 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:25.638263 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.638878 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.640283 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.640845 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.641051 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.641106 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.641342 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.641424 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:34:25.641567 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.637250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:44817 (local address 127.27.122.195:43423)
0928 16:34:25.637571 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.637585 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.637704 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.637947 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.637953 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.637978 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:25.638138 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.638145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.638988 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.638994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.640158 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.640172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.641079 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.641083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.641185 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.641210 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.641317 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.641386 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":27,"thread_start_us":67,"threads_started":1}
I20260928 16:34:25.641963 30251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.637192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40857 (local address 127.27.122.195:60299)
0928 16:34:25.637424 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.637439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.637516 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.638038 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.638041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.638057 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:25.638175 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.638180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.639200 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.639204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.640437 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.640447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.641526 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.641530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.641654 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.641680 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.641764 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.641846 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":47}
I20260928 16:34:25.642549 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.637648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:60299 (local address 127.27.122.193:40857)
0928 16:34:25.637795 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.637800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.637813 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:34:25.637864 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.637866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.637931 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.638001 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:25.638299 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.639057 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.640594 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.641307 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.641424 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.641468 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.642397 (+ 929us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.642462 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:34:25.642495 30172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d601ec9bbae4810b9bd04c1d1bb391c" candidate_uuid: "c87c9eb089bd4074b9864cba03cda1bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccb8b455e8e4425b6bd303f8b721403" is_pre_election: true
I20260928 16:34:25.642692 30172 raft_consensus.cc:2454] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87c9eb089bd4074b9864cba03cda1bd in term 0.
I20260928 16:34:25.643059 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d601ec9bbae4810b9bd04c1d1bb391c" candidate_uuid: "c87c9eb089bd4074b9864cba03cda1bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e132429f6d4c5690730ec900474d9c" is_pre_election: true
I20260928 16:34:25.643323 30032 raft_consensus.cc:2454] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87c9eb089bd4074b9864cba03cda1bd in term 0.
I20260928 16:34:25.643784 30245 leader_election.cc:302] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c87c9eb089bd4074b9864cba03cda1bd, dccb8b455e8e4425b6bd303f8b721403; no voters:
I20260928 16:34:25.644045 30389 raft_consensus.cc:2781] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:25.644122 30389 raft_consensus.cc:482] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:25.644169 30389 raft_consensus.cc:3037] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:25.645264 30389 raft_consensus.cc:504] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.645632 30389 leader_election.cc:288] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [CANDIDATE]: Term 1 election: Requested vote from peers 00e132429f6d4c5690730ec900474d9c (127.27.122.193:40857), dccb8b455e8e4425b6bd303f8b721403 (127.27.122.194:44817)
I20260928 16:34:25.645924 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d601ec9bbae4810b9bd04c1d1bb391c" candidate_uuid: "c87c9eb089bd4074b9864cba03cda1bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e132429f6d4c5690730ec900474d9c"
I20260928 16:34:25.645982 30172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d601ec9bbae4810b9bd04c1d1bb391c" candidate_uuid: "c87c9eb089bd4074b9864cba03cda1bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dccb8b455e8e4425b6bd303f8b721403"
I20260928 16:34:25.646066 30032 raft_consensus.cc:3037] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:25.646100 30172 raft_consensus.cc:3037] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:25.646960 30032 raft_consensus.cc:2454] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87c9eb089bd4074b9864cba03cda1bd in term 1.
I20260928 16:34:25.647229 30172 raft_consensus.cc:2454] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87c9eb089bd4074b9864cba03cda1bd in term 1.
I20260928 16:34:25.647464 30243 leader_election.cc:302] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e132429f6d4c5690730ec900474d9c, c87c9eb089bd4074b9864cba03cda1bd; no voters:
I20260928 16:34:25.647866 30389 raft_consensus.cc:2781] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:25.648028 30389 raft_consensus.cc:686] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [term 1 LEADER]: Becoming Leader. State: Replica: c87c9eb089bd4074b9864cba03cda1bd, State: Running, Role: LEADER
I20260928 16:34:25.648290 30389 consensus_queue.cc:237] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:25.650785 29873 catalog_manager.cc:5495] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd reported cstate change: term changed from 0 to 1, leader changed from <none> to c87c9eb089bd4074b9864cba03cda1bd (127.27.122.195). New cstate: current_term: 1 leader_uuid: "c87c9eb089bd4074b9864cba03cda1bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } health_report { overall_health: HEALTHY } } }
I20260928 16:34:25.670784 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.666864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.27.122.195:46879)
0928 16:34:25.666975 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.666982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.667001 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:34:25.667248 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.667250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.667334 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.667452 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:25.667713 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.668271 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.669260 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.669503 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.669632 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.669907 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:25.670377 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:25.670511 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.670576 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.670638 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:34:25.682296 30364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.678577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40857 (local address 127.27.122.195:51483)
0928 16:34:25.678831 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.678845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.678925 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.679193 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.679195 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.679201 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:34:25.679309 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.679313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.680297 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.680299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.680881 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.680888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.681514 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.681516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.681593 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.681637 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:34:25.681968 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:34:25.681978 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.682046 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.682150 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260928 16:34:25.682309 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.678709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:51483 (local address 127.27.122.193:40857)
0928 16:34:25.678850 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.678855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.678887 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:34:25.679019 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.679022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.679081 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.679176 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:25.679411 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.680191 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.681044 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.681397 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.681499 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.681762 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:25.681849 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:25.681923 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.682142 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.682198 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:34:25.688882 30172 raft_consensus.cc:1261] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Refusing update from remote peer c87c9eb089bd4074b9864cba03cda1bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:25.688989 30032 raft_consensus.cc:1261] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Refusing update from remote peer c87c9eb089bd4074b9864cba03cda1bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:25.689570 30431 consensus_queue.cc:1035] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:25.689949 30431 consensus_queue.cc:1035] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:25.708218 30380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.704454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:40305 (local address 127.27.122.195:46879)
0928 16:34:25.704653 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:25.704657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:25.704878 (+ 221us) server_negotiation.cc:407] Connection header received
0928 16:34:25.705051 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:25.705054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:25.705115 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:25.705207 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:25.705612 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.706093 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.707132 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.707364 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.707500 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.707758 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:25.707841 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:25.707917 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:25.708031 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 16:34:25.708091 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 16:34:25.708235 30419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:25.704366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:46879 (local address 127.27.122.193:40305)
0928 16:34:25.704811 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:25.704826 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:25.704939 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:25.705218 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:25.705221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:25.705230 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:34:25.705334 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.705341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.706210 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.706214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.706990 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:25.707000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:25.707506 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:25.707509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:25.707591 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:25.707623 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:34:25.707926 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:34:25.707934 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:34:25.708007 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 16:34:25.708098 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349}
W20260928 16:34:25.709103 30409 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 16:34:27.269263 29873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:27.269420 29873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:27.271688 29873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6: Sending DeleteTablet for 3 replicas of tablet 0d601ec9bbae4810b9bd04c1d1bb391c
I20260928 16:34:27.272192 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d601ec9bbae4810b9bd04c1d1bb391c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:27 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:55220
I20260928 16:34:27.272210 30152 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d601ec9bbae4810b9bd04c1d1bb391c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:27 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:52256
I20260928 16:34:27.272287 30292 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d601ec9bbae4810b9bd04c1d1bb391c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:27 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:48420
I20260928 16:34:27.272501 30462 tablet_replica.cc:330] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c: stopping tablet replica
I20260928 16:34:27.272634 30462 raft_consensus.cc:2229] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:27.272643 30461 tablet_replica.cc:330] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403: stopping tablet replica
I20260928 16:34:27.272647 30463 tablet_replica.cc:330] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd: stopping tablet replica
I20260928 16:34:27.272742 30462 raft_consensus.cc:2258] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:27.272882 30463 raft_consensus.cc:2229] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:27.273240 30463 raft_consensus.cc:2258] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:27.273381 30461 raft_consensus.cc:2229] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:27.273453 30462 ts_tablet_manager.cc:1905] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:27.273497 30461 raft_consensus.cc:2258] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:27.274012 30463 ts_tablet_manager.cc:1905] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:27.274019 30461 ts_tablet_manager.cc:1905] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:27.276546 30462 ts_tablet_manager.cc:1918] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:34:27.276587 30461 ts_tablet_manager.cc:1918] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:34:27.276648 30462 log.cc:1196] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/wal/wals/0d601ec9bbae4810b9bd04c1d1bb391c
I20260928 16:34:27.276662 30461 log.cc:1196] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/0d601ec9bbae4810b9bd04c1d1bb391c
I20260928 16:34:27.276710 30463 ts_tablet_manager.cc:1918] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:34:27.276786 30463 log.cc:1196] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/0d601ec9bbae4810b9bd04c1d1bb391c
I20260928 16:34:27.276963 30462 ts_tablet_manager.cc:1939] T 0d601ec9bbae4810b9bd04c1d1bb391c P 00e132429f6d4c5690730ec900474d9c: Deleting consensus metadata
I20260928 16:34:27.277112 30463 ts_tablet_manager.cc:1939] T 0d601ec9bbae4810b9bd04c1d1bb391c P c87c9eb089bd4074b9864cba03cda1bd: Deleting consensus metadata
I20260928 16:34:27.277271 30461 ts_tablet_manager.cc:1939] T 0d601ec9bbae4810b9bd04c1d1bb391c P dccb8b455e8e4425b6bd303f8b721403: Deleting consensus metadata
I20260928 16:34:27.277681 29858 catalog_manager.cc:4841] TS 00e132429f6d4c5690730ec900474d9c (127.27.122.193:40857): tablet 0d601ec9bbae4810b9bd04c1d1bb391c (table test-table [id=0afbfd8be52043bab1890c98f457798b]) successfully deleted
I20260928 16:34:27.277801 29859 catalog_manager.cc:4841] TS c87c9eb089bd4074b9864cba03cda1bd (127.27.122.195:46879): tablet 0d601ec9bbae4810b9bd04c1d1bb391c (table test-table [id=0afbfd8be52043bab1890c98f457798b]) successfully deleted
I20260928 16:34:27.277983 29860 catalog_manager.cc:4841] TS dccb8b455e8e4425b6bd303f8b721403 (127.27.122.194:44817): tablet 0d601ec9bbae4810b9bd04c1d1bb391c (table test-table [id=0afbfd8be52043bab1890c98f457798b]) successfully deleted
Sep 28 16:34:27 dist-test-slave-g9sh krb5kdc[29829](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613267, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:34:27 dist-test-slave-g9sh krb5kdc[29829](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613267, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:27.305424 30471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:27.292706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34044 (local address 127.27.122.254:39421)
0928 16:34:27.293032 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:27.293039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:27.293054 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:34:27.293193 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:27.293195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:27.293240 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:27.293319 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:27.293943 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.294485 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.295251 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.295470 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.295542 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:27.302124 (+ 6582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:27.302127 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:27.302129 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:27.302154 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:27.303677 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:27.304113 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:27.304117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:27.304120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:27.304205 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:27.304489 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:27.304493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:27.304494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:27.304685 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:27.304827 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:27.305120 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 16:34:27.305251 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":114,"threads_started":1}
I20260928 16:34:27.307791 29873 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:34:27.308177 29873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:27.312709 30010 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8294d8d9934935be791b72ec1c66c5 (DEFAULT_TABLE table=test-table [id=7465526800f748a7b3f74548a6d5262e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:27.312754 30152 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8294d8d9934935be791b72ec1c66c5 (DEFAULT_TABLE table=test-table [id=7465526800f748a7b3f74548a6d5262e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:27.312861 30292 tablet_service.cc:1467] Processing CreateTablet for tablet 0b8294d8d9934935be791b72ec1c66c5 (DEFAULT_TABLE table=test-table [id=7465526800f748a7b3f74548a6d5262e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:27.313071 30152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8294d8d9934935be791b72ec1c66c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:27.313071 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8294d8d9934935be791b72ec1c66c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:27.313149 30292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b8294d8d9934935be791b72ec1c66c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:27.315361 30472 tablet_bootstrap.cc:492] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c: Bootstrap starting.
I20260928 16:34:27.316012 30474 tablet_bootstrap.cc:492] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd: Bootstrap starting.
I20260928 16:34:27.316373 30472 tablet_bootstrap.cc:654] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:27.316864 30474 tablet_bootstrap.cc:654] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:27.317091 30473 tablet_bootstrap.cc:492] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403: Bootstrap starting.
I20260928 16:34:27.317483 30472 tablet_bootstrap.cc:492] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c: No bootstrap required, opened a new log
I20260928 16:34:27.317554 30472 ts_tablet_manager.cc:1397] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:27.317967 30472 raft_consensus.cc:348] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.318086 30472 raft_consensus.cc:374] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:27.318128 30472 raft_consensus.cc:729] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e132429f6d4c5690730ec900474d9c, State: Initialized, Role: FOLLOWER
I20260928 16:34:27.318325 30474 tablet_bootstrap.cc:492] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd: No bootstrap required, opened a new log
I20260928 16:34:27.318285 30472 consensus_queue.cc:260] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.318418 30474 ts_tablet_manager.cc:1397] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 16:34:27.318502 30473 tablet_bootstrap.cc:654] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:27.318845 30474 raft_consensus.cc:348] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.318965 30474 raft_consensus.cc:374] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:27.319002 30474 raft_consensus.cc:729] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87c9eb089bd4074b9864cba03cda1bd, State: Initialized, Role: FOLLOWER
I20260928 16:34:27.319048 30472 ts_tablet_manager.cc:1428] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:27.319136 30474 consensus_queue.cc:260] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.319396 30474 ts_tablet_manager.cc:1428] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:34:27.320035 30473 tablet_bootstrap.cc:492] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403: No bootstrap required, opened a new log
I20260928 16:34:27.320130 30473 ts_tablet_manager.cc:1397] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 16:34:27.320791 30473 raft_consensus.cc:348] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.320951 30473 raft_consensus.cc:374] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:27.320995 30473 raft_consensus.cc:729] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccb8b455e8e4425b6bd303f8b721403, State: Initialized, Role: FOLLOWER
I20260928 16:34:27.321159 30473 consensus_queue.cc:260] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.321610 30473 ts_tablet_manager.cc:1428] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260928 16:34:27.393786 30478 raft_consensus.cc:482] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:27.393939 30478 raft_consensus.cc:504] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.395169 30478 leader_election.cc:288] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00e132429f6d4c5690730ec900474d9c (127.27.122.193:40857), c87c9eb089bd4074b9864cba03cda1bd (127.27.122.195:46879)
I20260928 16:34:27.399781 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:27.395770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:32937 (local address 127.27.122.193:40857)
0928 16:34:27.396077 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:27.396086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:27.396098 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:34:27.396143 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:27.396145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:27.396183 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:27.396261 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:27.396639 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.397480 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.398591 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.399218 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.399321 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:27.399378 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:27.399566 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 16:34:27.399640 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":97,"threads_started":1}
I20260928 16:34:27.399781 30481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:27.395482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:48373 (local address 127.27.122.195:46879)
0928 16:34:27.396087 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:27.396102 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:27.396114 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:34:27.396159 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:27.396161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:27.396198 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:27.396301 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:27.396670 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.397292 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.398464 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.399139 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.399276 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:27.399334 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:27.399561 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 16:34:27.399642 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":140,"threads_started":1}
I20260928 16:34:27.399786 30480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:27.395362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40857 (local address 127.27.122.194:32937)
0928 16:34:27.395862 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:27.395877 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:27.395979 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:27.396309 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:27.396312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:27.396328 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:27.396512 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:27.396518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.397599 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.397602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:27.398474 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:27.398484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.399315 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.399318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:27.399427 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:27.399443 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 16:34:27.399526 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 16:34:27.399592 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":145,"spinlock_wait_cycles":9984,"thread_start_us":93,"threads_started":1}
I20260928 16:34:27.399796 30479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:27.395362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:46879 (local address 127.27.122.194:48373)
0928 16:34:27.395815 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:27.395833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:27.395945 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:27.396335 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:27.396338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:27.396385 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:27.396519 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:27.396524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.397404 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.397408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:27.398313 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:27.398323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.399290 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.399297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:27.399395 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:27.399425 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:34:27.399508 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 16:34:27.399586 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":4,"spinlock_wait_cycles":3584,"thread_start_us":182,"threads_started":1}
I20260928 16:34:27.400230 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8294d8d9934935be791b72ec1c66c5" candidate_uuid: "dccb8b455e8e4425b6bd303f8b721403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e132429f6d4c5690730ec900474d9c" is_pre_election: true
I20260928 16:34:27.400388 30032 raft_consensus.cc:2454] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dccb8b455e8e4425b6bd303f8b721403 in term 0.
I20260928 16:34:27.400754 30312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8294d8d9934935be791b72ec1c66c5" candidate_uuid: "dccb8b455e8e4425b6bd303f8b721403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87c9eb089bd4074b9864cba03cda1bd" is_pre_election: true
I20260928 16:34:27.400897 30312 raft_consensus.cc:2454] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dccb8b455e8e4425b6bd303f8b721403 in term 0.
I20260928 16:34:27.401158 30103 leader_election.cc:302] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e132429f6d4c5690730ec900474d9c, dccb8b455e8e4425b6bd303f8b721403; no voters:
I20260928 16:34:27.401439 30478 raft_consensus.cc:2781] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:27.401513 30478 raft_consensus.cc:482] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:27.401542 30478 raft_consensus.cc:3037] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:27.402554 30478 raft_consensus.cc:504] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.402912 30478 leader_election.cc:288] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [CANDIDATE]: Term 1 election: Requested vote from peers 00e132429f6d4c5690730ec900474d9c (127.27.122.193:40857), c87c9eb089bd4074b9864cba03cda1bd (127.27.122.195:46879)
I20260928 16:34:27.403275 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8294d8d9934935be791b72ec1c66c5" candidate_uuid: "dccb8b455e8e4425b6bd303f8b721403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e132429f6d4c5690730ec900474d9c"
I20260928 16:34:27.403259 30312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b8294d8d9934935be791b72ec1c66c5" candidate_uuid: "dccb8b455e8e4425b6bd303f8b721403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87c9eb089bd4074b9864cba03cda1bd"
I20260928 16:34:27.403414 30032 raft_consensus.cc:3037] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:27.403425 30312 raft_consensus.cc:3037] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:27.404311 30312 raft_consensus.cc:2454] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dccb8b455e8e4425b6bd303f8b721403 in term 1.
I20260928 16:34:27.404326 30032 raft_consensus.cc:2454] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dccb8b455e8e4425b6bd303f8b721403 in term 1.
I20260928 16:34:27.404608 30103 leader_election.cc:302] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e132429f6d4c5690730ec900474d9c, dccb8b455e8e4425b6bd303f8b721403; no voters:
I20260928 16:34:27.404837 30478 raft_consensus.cc:2781] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:27.405067 30478 raft_consensus.cc:686] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [term 1 LEADER]: Becoming Leader. State: Replica: dccb8b455e8e4425b6bd303f8b721403, State: Running, Role: LEADER
I20260928 16:34:27.405334 30478 consensus_queue.cc:237] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.407785 29874 catalog_manager.cc:5495] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 reported cstate change: term changed from 0 to 1, leader changed from <none> to dccb8b455e8e4425b6bd303f8b721403 (127.27.122.194). New cstate: current_term: 1 leader_uuid: "dccb8b455e8e4425b6bd303f8b721403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } health_report { overall_health: UNKNOWN } } }
I20260928 16:34:27.427260 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:27.423344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.27.122.194:44817)
0928 16:34:27.423668 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:27.423677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:27.423697 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:34:27.423762 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:27.423763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:27.423811 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:27.423938 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:27.424193 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.424803 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.425740 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.425976 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.426114 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:27.426392 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:27.426847 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:27.426982 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:27.427040 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 16:34:27.427100 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":152,"threads_started":1}
I20260928 16:34:27.436667 30482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:27.433730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:56451 (local address 127.27.122.193:40857)
0928 16:34:27.433840 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:27.433843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:27.433990 (+ 147us) server_negotiation.cc:407] Connection header received
0928 16:34:27.434090 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:27.434092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:27.434131 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:27.434195 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:27.434397 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.434908 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.435725 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.435932 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.436049 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:27.436298 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:27.436371 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:27.436428 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:27.436516 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 16:34:27.436572 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:34:27.436673 30489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:27.433509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40857 (local address 127.27.122.194:56451)
0928 16:34:27.433921 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:27.433933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:27.434014 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:27.434218 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:27.434220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:27.434226 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:34:27.434314 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:27.434318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.435002 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.435004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:27.435622 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:27.435630 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.436034 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.436037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:27.436121 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:27.436160 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:34:27.436449 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:34:27.436454 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:34:27.436501 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 16:34:27.436561 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":124,"threads_started":1}
I20260928 16:34:27.442193 30032 raft_consensus.cc:1261] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Refusing update from remote peer dccb8b455e8e4425b6bd303f8b721403: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:27.442193 30312 raft_consensus.cc:1261] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Refusing update from remote peer dccb8b455e8e4425b6bd303f8b721403: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:27.442684 30478 consensus_queue.cc:1035] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:27.442889 30483 consensus_queue.cc:1035] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:27.459915 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:27.456522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:57009 (local address 127.27.122.194:44817)
0928 16:34:27.456647 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:27.456652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:27.456665 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:34:27.456797 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:27.456801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:27.456863 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:27.456970 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:27.457223 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.457790 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.458783 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.459059 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.459187 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:27.459454 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:27.459536 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:27.459631 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:27.459722 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 16:34:27.459783 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:34:27.459946 30495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:27.456269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:44817 (local address 127.27.122.193:57009)
0928 16:34:27.456519 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:27.456533 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:27.456663 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:27.456985 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:27.456988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:27.457000 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:34:27.457112 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:27.457117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.457882 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.457884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:27.458668 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:27.458680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:27.459204 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:27.459207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:27.459296 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:27.459327 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:34:27.459651 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:34:27.459657 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:34:27.459710 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 16:34:27.459777 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":115,"threads_started":1}
I20260928 16:34:27.482223 29874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:27.482362 29874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:27.484319 29874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6: Sending DeleteTablet for 3 replicas of tablet 0b8294d8d9934935be791b72ec1c66c5
I20260928 16:34:27.484819 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b8294d8d9934935be791b72ec1c66c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:27 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:55220
I20260928 16:34:27.484988 30152 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b8294d8d9934935be791b72ec1c66c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:27 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:52256
I20260928 16:34:27.484982 30292 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b8294d8d9934935be791b72ec1c66c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:27 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:48420
I20260928 16:34:27.485018 30462 tablet_replica.cc:330] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c: stopping tablet replica
I20260928 16:34:27.485137 30462 raft_consensus.cc:2229] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:27.485168 30461 tablet_replica.cc:330] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403: stopping tablet replica
I20260928 16:34:27.485234 30462 raft_consensus.cc:2258] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:27.485275 30461 raft_consensus.cc:2229] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:27.485575 30461 raft_consensus.cc:2258] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:27.485764 30463 tablet_replica.cc:330] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd: stopping tablet replica
I20260928 16:34:27.485875 30463 raft_consensus.cc:2229] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:27.485972 30463 raft_consensus.cc:2258] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:27.486105 30462 ts_tablet_manager.cc:1905] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:27.486724 30463 ts_tablet_manager.cc:1905] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:27.487422 29874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:34:27.487828 29874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:27.489645 30462 ts_tablet_manager.cc:1918] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:27.489725 30462 log.cc:1196] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-0/wal/wals/0b8294d8d9934935be791b72ec1c66c5
I20260928 16:34:27.489856 30461 ts_tablet_manager.cc:1905] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:27.489998 30462 ts_tablet_manager.cc:1939] T 0b8294d8d9934935be791b72ec1c66c5 P 00e132429f6d4c5690730ec900474d9c: Deleting consensus metadata
I20260928 16:34:27.490018 30463 ts_tablet_manager.cc:1918] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:27.490108 30463 log.cc:1196] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/0b8294d8d9934935be791b72ec1c66c5
I20260928 16:34:27.490347 30463 ts_tablet_manager.cc:1939] T 0b8294d8d9934935be791b72ec1c66c5 P c87c9eb089bd4074b9864cba03cda1bd: Deleting consensus metadata
I20260928 16:34:27.490809 29858 catalog_manager.cc:4841] TS 00e132429f6d4c5690730ec900474d9c (127.27.122.193:40857): tablet 0b8294d8d9934935be791b72ec1c66c5 (table test-table [id=7465526800f748a7b3f74548a6d5262e]) successfully deleted
I20260928 16:34:27.490926 29859 catalog_manager.cc:4841] TS c87c9eb089bd4074b9864cba03cda1bd (127.27.122.195:46879): tablet 0b8294d8d9934935be791b72ec1c66c5 (table test-table [id=7465526800f748a7b3f74548a6d5262e]) successfully deleted
I20260928 16:34:27.492328 30461 ts_tablet_manager.cc:1918] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:27.492393 30461 log.cc:1196] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/0b8294d8d9934935be791b72ec1c66c5
I20260928 16:34:27.492640 30461 ts_tablet_manager.cc:1939] T 0b8294d8d9934935be791b72ec1c66c5 P dccb8b455e8e4425b6bd303f8b721403: Deleting consensus metadata
I20260928 16:34:27.493281 29860 catalog_manager.cc:4841] TS dccb8b455e8e4425b6bd303f8b721403 (127.27.122.194:44817): tablet 0b8294d8d9934935be791b72ec1c66c5 (table test-table [id=7465526800f748a7b3f74548a6d5262e]) successfully deleted
I20260928 16:34:27.496032 30010 tablet_service.cc:1467] Processing CreateTablet for tablet c658d71366b243e7ae2a6fa78784662c (DEFAULT_TABLE table=test-table [id=26252ce3dba748ae8fc55c33885284f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:27.496145 30152 tablet_service.cc:1467] Processing CreateTablet for tablet c658d71366b243e7ae2a6fa78784662c (DEFAULT_TABLE table=test-table [id=26252ce3dba748ae8fc55c33885284f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:27.496289 30010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c658d71366b243e7ae2a6fa78784662c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:27.496260 30292 tablet_service.cc:1467] Processing CreateTablet for tablet c658d71366b243e7ae2a6fa78784662c (DEFAULT_TABLE table=test-table [id=26252ce3dba748ae8fc55c33885284f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:27.496397 30152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c658d71366b243e7ae2a6fa78784662c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:27.496479 30292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c658d71366b243e7ae2a6fa78784662c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:27.498627 30473 tablet_bootstrap.cc:492] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403: Bootstrap starting.
I20260928 16:34:27.498868 30474 tablet_bootstrap.cc:492] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd: Bootstrap starting.
I20260928 16:34:27.499593 30473 tablet_bootstrap.cc:654] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:27.499742 30474 tablet_bootstrap.cc:654] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:27.500386 30472 tablet_bootstrap.cc:492] T c658d71366b243e7ae2a6fa78784662c P 00e132429f6d4c5690730ec900474d9c: Bootstrap starting.
I20260928 16:34:27.501210 30472 tablet_bootstrap.cc:654] T c658d71366b243e7ae2a6fa78784662c P 00e132429f6d4c5690730ec900474d9c: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:27.501247 30474 tablet_bootstrap.cc:492] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd: No bootstrap required, opened a new log
I20260928 16:34:27.501319 30474 ts_tablet_manager.cc:1397] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 16:34:27.501662 30474 raft_consensus.cc:348] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.501765 30474 raft_consensus.cc:374] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:27.501792 30474 raft_consensus.cc:729] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87c9eb089bd4074b9864cba03cda1bd, State: Initialized, Role: FOLLOWER
I20260928 16:34:27.501895 30474 consensus_queue.cc:260] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.502028 30473 tablet_bootstrap.cc:492] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403: No bootstrap required, opened a new log
I20260928 16:34:27.502108 30473 ts_tablet_manager.cc:1397] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 16:34:27.502118 30474 ts_tablet_manager.cc:1428] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:34:27.502508 30473 raft_consensus.cc:348] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.502612 30473 raft_consensus.cc:374] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:27.502643 30473 raft_consensus.cc:729] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccb8b455e8e4425b6bd303f8b721403, State: Initialized, Role: FOLLOWER
I20260928 16:34:27.502789 30473 consensus_queue.cc:260] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.503028 30473 ts_tablet_manager.cc:1428] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:34:27.506765 30472 tablet_bootstrap.cc:492] T c658d71366b243e7ae2a6fa78784662c P 00e132429f6d4c5690730ec900474d9c: No bootstrap required, opened a new log
I20260928 16:34:27.506850 30472 ts_tablet_manager.cc:1397] T c658d71366b243e7ae2a6fa78784662c P 00e132429f6d4c5690730ec900474d9c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260928 16:34:27.507221 30472 raft_consensus.cc:348] T c658d71366b243e7ae2a6fa78784662c P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.507332 30472 raft_consensus.cc:374] T c658d71366b243e7ae2a6fa78784662c P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:27.507370 30472 raft_consensus.cc:729] T c658d71366b243e7ae2a6fa78784662c P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e132429f6d4c5690730ec900474d9c, State: Initialized, Role: FOLLOWER
I20260928 16:34:27.507485 30472 consensus_queue.cc:260] T c658d71366b243e7ae2a6fa78784662c P 00e132429f6d4c5690730ec900474d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.507710 30472 ts_tablet_manager.cc:1428] T c658d71366b243e7ae2a6fa78784662c P 00e132429f6d4c5690730ec900474d9c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:34:27.526235 30492 raft_consensus.cc:482] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:27.526335 30492 raft_consensus.cc:504] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.526782 30492 leader_election.cc:288] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00e132429f6d4c5690730ec900474d9c (127.27.122.193:40857), c87c9eb089bd4074b9864cba03cda1bd (127.27.122.195:46879)
I20260928 16:34:27.527096 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c658d71366b243e7ae2a6fa78784662c" candidate_uuid: "dccb8b455e8e4425b6bd303f8b721403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e132429f6d4c5690730ec900474d9c" is_pre_election: true
I20260928 16:34:27.527187 30311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c658d71366b243e7ae2a6fa78784662c" candidate_uuid: "dccb8b455e8e4425b6bd303f8b721403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87c9eb089bd4074b9864cba03cda1bd" is_pre_election: true
I20260928 16:34:27.527274 30032 raft_consensus.cc:2454] T c658d71366b243e7ae2a6fa78784662c P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dccb8b455e8e4425b6bd303f8b721403 in term 0.
I20260928 16:34:27.527370 30311 raft_consensus.cc:2454] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dccb8b455e8e4425b6bd303f8b721403 in term 0.
I20260928 16:34:27.527597 30103 leader_election.cc:302] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e132429f6d4c5690730ec900474d9c, dccb8b455e8e4425b6bd303f8b721403; no voters:
I20260928 16:34:27.527863 30492 raft_consensus.cc:2781] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:27.527932 30492 raft_consensus.cc:482] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:27.527985 30492 raft_consensus.cc:3037] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:27.529088 30492 raft_consensus.cc:504] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.529712 30032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c658d71366b243e7ae2a6fa78784662c" candidate_uuid: "dccb8b455e8e4425b6bd303f8b721403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00e132429f6d4c5690730ec900474d9c"
I20260928 16:34:27.529881 30032 raft_consensus.cc:3037] T c658d71366b243e7ae2a6fa78784662c P 00e132429f6d4c5690730ec900474d9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:27.530011 30311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c658d71366b243e7ae2a6fa78784662c" candidate_uuid: "dccb8b455e8e4425b6bd303f8b721403" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87c9eb089bd4074b9864cba03cda1bd"
I20260928 16:34:27.530123 30311 raft_consensus.cc:3037] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:27.530764 30032 raft_consensus.cc:2454] T c658d71366b243e7ae2a6fa78784662c P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dccb8b455e8e4425b6bd303f8b721403 in term 1.
I20260928 16:34:27.529623 30492 leader_election.cc:288] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [CANDIDATE]: Term 1 election: Requested vote from peers 00e132429f6d4c5690730ec900474d9c (127.27.122.193:40857), c87c9eb089bd4074b9864cba03cda1bd (127.27.122.195:46879)
I20260928 16:34:27.531106 30103 leader_election.cc:302] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00e132429f6d4c5690730ec900474d9c, dccb8b455e8e4425b6bd303f8b721403; no voters:
I20260928 16:34:27.531296 30492 raft_consensus.cc:2781] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:27.531427 30492 raft_consensus.cc:686] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [term 1 LEADER]: Becoming Leader. State: Replica: dccb8b455e8e4425b6bd303f8b721403, State: Running, Role: LEADER
I20260928 16:34:27.531488 30311 raft_consensus.cc:2454] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dccb8b455e8e4425b6bd303f8b721403 in term 1.
I20260928 16:34:27.531570 30492 consensus_queue.cc:237] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } }
I20260928 16:34:27.533326 29874 catalog_manager.cc:5495] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 reported cstate change: term changed from 0 to 1, leader changed from <none> to dccb8b455e8e4425b6bd303f8b721403 (127.27.122.194). New cstate: current_term: 1 leader_uuid: "dccb8b455e8e4425b6bd303f8b721403" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dccb8b455e8e4425b6bd303f8b721403" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 } health_report { overall_health: UNKNOWN } } }
I20260928 16:34:27.549916 30032 raft_consensus.cc:1261] T c658d71366b243e7ae2a6fa78784662c P 00e132429f6d4c5690730ec900474d9c [term 1 FOLLOWER]: Refusing update from remote peer dccb8b455e8e4425b6bd303f8b721403: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:27.550132 30312 raft_consensus.cc:1261] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Refusing update from remote peer dccb8b455e8e4425b6bd303f8b721403: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:27.550336 30478 consensus_queue.cc:1035] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00e132429f6d4c5690730ec900474d9c" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:27.550598 30478 consensus_queue.cc:1035] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87c9eb089bd4074b9864cba03cda1bd" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:34:27.565155 30444 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 16:34:29.115939 29873 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:29.116119 29873 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:29.118528 29873 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cadd6bb2214c4ab4bb3fb10c5aef7ed6: Sending DeleteTablet for 3 replicas of tablet c658d71366b243e7ae2a6fa78784662c
I20260928 16:34:29.119056 30152 tablet_service.cc:1514] Processing DeleteTablet for tablet c658d71366b243e7ae2a6fa78784662c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:29 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:52256
I20260928 16:34:29.119084 30010 tablet_service.cc:1514] Processing DeleteTablet for tablet c658d71366b243e7ae2a6fa78784662c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:29 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:55220
I20260928 16:34:29.119191 30292 tablet_service.cc:1514] Processing DeleteTablet for tablet c658d71366b243e7ae2a6fa78784662c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:29 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:48420
I20260928 16:34:29.119446 30526 tablet_replica.cc:330] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403: stopping tablet replica
I20260928 16:34:29.119488 30527 tablet_replica.cc:330] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd: stopping tablet replica
I20260928 16:34:29.119577 30526 raft_consensus.cc:2229] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:29.119608 30527 raft_consensus.cc:2229] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:29.119707 30527 raft_consensus.cc:2258] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:29.119763 30526 raft_consensus.cc:2258] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:29.119943 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 29946
I20260928 16:34:29.121110 30526 ts_tablet_manager.cc:1905] T c658d71366b243e7ae2a6fa78784662c P dccb8b455e8e4425b6bd303f8b721403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:29.121805 30527 ts_tablet_manager.cc:1905] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 16:34:29.131564 29858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:40857 (error 108)
I20260928 16:34:29.131749 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 30087
I20260928 16:34:29.134184 30529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:29.133792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40857 (local address 127.0.0.1:55254)
0928 16:34:29.134029 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:29.134095 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":84,"threads_started":1}
W20260928 16:34:29.134321 29858 catalog_manager.cc:4584] TS 00e132429f6d4c5690730ec900474d9c (127.27.122.193:40857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c658d71366b243e7ae2a6fa78784662c: Network error: Client connection negotiation failed: client connection to 127.27.122.193:40857: connect: Connection refused (error 111)
I20260928 16:34:29.187728 30529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:29.187405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40857 (local address 127.0.0.1:55264)
0928 16:34:29.187585 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:29.187647 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 16:34:29.188598 30529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:29.188333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40857 (local address 127.0.0.1:55268)
0928 16:34:29.188475 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:29.188536 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 16:34:29.198761 30527 ts_tablet_manager.cc:1918] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 16:34:29.198871 30527 log.cc:1196] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/c658d71366b243e7ae2a6fa78784662c
I20260928 16:34:29.199003 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 30227
I20260928 16:34:29.199251 30527 ts_tablet_manager.cc:1939] T c658d71366b243e7ae2a6fa78784662c P c87c9eb089bd4074b9864cba03cda1bd: Deleting consensus metadata
I20260928 16:34:29.200650 30529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:29.199115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:44817 (local address 127.0.0.1:52274)
0928 16:34:29.200476 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:29.200543 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.194:44817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1264}
I20260928 16:34:29.206670 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 29842
2026-09-28T16:34:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6054 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:31 dist-test-slave-g9sh krb5kdc[30532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:31 dist-test-slave-g9sh krb5kdc[30532](info): set up 2 sockets
Sep 28 16:34:31 dist-test-slave-g9sh krb5kdc[30532](info): commencing operation
krb5kdc: starting...
W20260928 16:34:31.991505 28139 mini_kdc.cc:121] Time spent starting KDC: real 2.761s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:32 dist-test-slave-g9sh krb5kdc[30532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:34:32.111857 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:43551
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:43551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:32.224381 30545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:32.224655 30545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:32.224695 30545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:32.228621 30545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:32.228685 30545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:32.228705 30545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:32.228722 30545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:32.228737 30545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:32.234018 30545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:43551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:43551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:32.235047 30545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:32.236052 30545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:34:32.245054 30545 server_base.cc:1034] running on GCE node
W20260928 16:34:32.244999 30553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:32.245002 30550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:32.245034 30551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:32.245793 30545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:32.246943 30545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:32.248143 30545 hybrid_clock.cc:648] HybridClock initialized: now 1790613272248123 us; error 26 us; skew 500 ppm
Sep 28 16:34:32 dist-test-slave-g9sh krb5kdc[30532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613272, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:32.250919 30545 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:34:32.251847 30545 webserver.cc:460] Webserver started at http://127.27.122.254:44333/ using document root <none> and password file <none>
I20260928 16:34:32.252414 30545 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:32.252506 30545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:32.252746 30545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:32.254801 30545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "b8367a8f51844051b5ea03137f7ae562"
format_stamp: "Formatted at 2026-09-28 16:34:32 on dist-test-slave-g9sh"
server_key: "17ef01f29d75b68a5c30144a654cb047"
server_key_iv: "67055f1dae4e03c2cbb8eea60ca0d193"
server_key_version: "encryptionkey@0"
I20260928 16:34:32.255359 30545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "b8367a8f51844051b5ea03137f7ae562"
format_stamp: "Formatted at 2026-09-28 16:34:32 on dist-test-slave-g9sh"
server_key: "17ef01f29d75b68a5c30144a654cb047"
server_key_iv: "67055f1dae4e03c2cbb8eea60ca0d193"
server_key_version: "encryptionkey@0"
I20260928 16:34:32.259033 30545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:34:32.261394 30560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:32.262611 30545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:32.262765 30545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "b8367a8f51844051b5ea03137f7ae562"
format_stamp: "Formatted at 2026-09-28 16:34:32 on dist-test-slave-g9sh"
server_key: "17ef01f29d75b68a5c30144a654cb047"
server_key_iv: "67055f1dae4e03c2cbb8eea60ca0d193"
server_key_version: "encryptionkey@0"
I20260928 16:34:32.262876 30545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:32.282153 30545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:32.285039 30545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:32.285243 30545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:32.292268 30545 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:43551
I20260928 16:34:32.292289 30612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:43551 every 8 connection(s)
I20260928 16:34:32.293217 30545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:32.295888 30613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:32.297916 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 30545
I20260928 16:34:32.298028 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:32.298298 28139 external_mini_cluster.cc:1428] Setting key 3dc52bd8b75f9ca0761a3e604f669a6d
I20260928 16:34:32.302418 30613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562: Bootstrap starting.
I20260928 16:34:32.304581 30613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:32.305418 30613 log.cc:824] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:32.307226 30613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562: No bootstrap required, opened a new log
I20260928 16:34:32.310622 30613 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8367a8f51844051b5ea03137f7ae562" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 43551 } }
I20260928 16:34:32.310829 30613 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:32.310873 30613 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8367a8f51844051b5ea03137f7ae562, State: Initialized, Role: FOLLOWER
I20260928 16:34:32.311293 30613 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8367a8f51844051b5ea03137f7ae562" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 43551 } }
I20260928 16:34:32.311416 30613 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:32.311461 30613 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:32.311529 30613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:32.312436 30613 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8367a8f51844051b5ea03137f7ae562" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 43551 } }
I20260928 16:34:32.312734 30613 leader_election.cc:302] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8367a8f51844051b5ea03137f7ae562; no voters:
I20260928 16:34:32.313040 30613 leader_election.cc:288] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:32.313158 30618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:32.313463 30618 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [term 1 LEADER]: Becoming Leader. State: Replica: b8367a8f51844051b5ea03137f7ae562, State: Running, Role: LEADER
I20260928 16:34:32.313902 30618 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8367a8f51844051b5ea03137f7ae562" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 43551 } }
I20260928 16:34:32.314432 30613 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 16:34:32 dist-test-slave-g9sh krb5kdc[30532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:32.315577 30619 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8367a8f51844051b5ea03137f7ae562" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8367a8f51844051b5ea03137f7ae562" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 43551 } } }
I20260928 16:34:32.315722 30619 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:32.315841 30620 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8367a8f51844051b5ea03137f7ae562. Latest consensus state: current_term: 1 leader_uuid: "b8367a8f51844051b5ea03137f7ae562" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8367a8f51844051b5ea03137f7ae562" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 43551 } } }
I20260928 16:34:32.315927 30620 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:32.316967 30627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:32.320077 30627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:32.321132 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:32.299768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.27.122.254:43551)
0928 16:34:32.300194 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:32.300208 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:32.300242 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:34:32.300883 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:32.300948 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:32.301265 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:32.301601 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:32.302409 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:32.303440 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:32.304293 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:32.304834 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:32.305051 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:32.316009 (+ 10958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:32.316016 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:32.316032 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:32.316064 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:32.318178 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:32.318657 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:32.318673 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:32.318679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:32.318798 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:32.319118 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:32.319122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:32.319123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:32.319518 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:32.319742 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:32.320036 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 16:34:32.320370 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":132,"threads_started":1}
I20260928 16:34:32.325839 30627 catalog_manager.cc:1347] Generated new cluster ID: 0556c63ddb284837b2686d72d960530e
I20260928 16:34:32.325904 30627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:32.345988 30627 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:32.346958 30627 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:32.351877 30627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8367a8f51844051b5ea03137f7ae562: Generated new TSK 0
I20260928 16:34:32.352567 30627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:34:32.392881 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:43551
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:32.531167 30639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:32.531428 30639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:32.531472 30639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:32.536079 30639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:32.536146 30639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:32.536208 30639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:32.541703 30639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:32.542721 30639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:32.543699 30639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:32.553258 30644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:32.553318 30639 server_base.cc:1034] running on GCE node
W20260928 16:34:32.553252 30645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:32.553252 30647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:32.554005 30639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:32.554615 30639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:32.555786 30639 hybrid_clock.cc:648] HybridClock initialized: now 1790613272555770 us; error 26 us; skew 500 ppm
Sep 28 16:34:32 dist-test-slave-g9sh krb5kdc[30532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613272, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:32.558651 30639 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:34:32.559515 30639 webserver.cc:460] Webserver started at http://127.27.122.193:39019/ using document root <none> and password file <none>
I20260928 16:34:32.560010 30639 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:32.560067 30639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:32.560226 30639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:32.562589 30639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "3a6b92009b414c7a8829d22e5cac9020"
format_stamp: "Formatted at 2026-09-28 16:34:32 on dist-test-slave-g9sh"
server_key: "7adba2b328d9be7152365dc59cc330eb"
server_key_iv: "83d1ed5ffcd280141267f557fb02fdb4"
server_key_version: "encryptionkey@0"
I20260928 16:34:32.563333 30639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "3a6b92009b414c7a8829d22e5cac9020"
format_stamp: "Formatted at 2026-09-28 16:34:32 on dist-test-slave-g9sh"
server_key: "7adba2b328d9be7152365dc59cc330eb"
server_key_iv: "83d1ed5ffcd280141267f557fb02fdb4"
server_key_version: "encryptionkey@0"
I20260928 16:34:32.567665 30639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 16:34:32.570187 30654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:32.571414 30639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:34:32.571548 30639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "3a6b92009b414c7a8829d22e5cac9020"
format_stamp: "Formatted at 2026-09-28 16:34:32 on dist-test-slave-g9sh"
server_key: "7adba2b328d9be7152365dc59cc330eb"
server_key_iv: "83d1ed5ffcd280141267f557fb02fdb4"
server_key_version: "encryptionkey@0"
I20260928 16:34:32.571657 30639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:32.590785 30639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:32.594045 30639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:32.594247 30639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:32.594838 30639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:32.595666 30639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:32.595746 30639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:32.595820 30639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:32.595870 30639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:32.604940 30639 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:34247
I20260928 16:34:32.604975 30767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:34247 every 8 connection(s)
I20260928 16:34:32.605805 30639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:32.609808 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 30639
I20260928 16:34:32.609941 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:34:32.610211 28139 external_mini_cluster.cc:1428] Setting key 50f1889902f3945b781c77efb6e91ac1
Sep 28 16:34:32 dist-test-slave-g9sh krb5kdc[30532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613272, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:34:32.620018 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:32.607569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:47711 (local address 127.27.122.254:43551)
0928 16:34:32.607701 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:32.607704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:32.608517 (+ 813us) server_negotiation.cc:407] Connection header received
0928 16:34:32.609769 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:32.609773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:32.609837 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:32.609942 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:32.611255 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:32.611950 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:32.613173 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:32.613415 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:32.613532 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:32.616317 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:32.616321 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:32.616323 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:32.616346 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:32.617945 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:32.618460 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:32.618466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:32.618467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:32.618555 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:32.618892 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:32.618897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:32.618899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:32.619075 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:32.619197 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:32.619724 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 16:34:32.619848 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:34:32.620850 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:32.607955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:43551 (local address 127.27.122.193:47711)
0928 16:34:32.608369 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:32.608411 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:32.609557 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:32.610140 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:32.610147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:32.610573 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:32.611112 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:32.611126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:32.612082 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:32.612085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:32.613044 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:32.613055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:32.613569 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:32.613573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:32.613751 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:32.614370 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:32.614394 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:32.616143 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:32.618070 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:32.618076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:32.618089 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:32.618348 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:32.618649 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:32.618653 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:32.618655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:32.618795 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:32.619195 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:32.619202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:32.619494 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 16:34:32.619680 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 16:34:32.619994 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
I20260928 16:34:32.622287 30768 heartbeater.cc:347] Connected to a master server at 127.27.122.254:43551
I20260928 16:34:32.622575 30768 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:32.623160 30768 heartbeater.cc:511] Master 127.27.122.254:43551 requested a full tablet report, sending...
I20260928 16:34:32.624805 30577 ts_manager.cc:194] Registered new tserver with Master: 3a6b92009b414c7a8829d22e5cac9020 (127.27.122.193:34247)
I20260928 16:34:32.626044 30577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:47711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:34:32.637821 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:43551
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:32.747098 30773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:32.747380 30773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:32.747463 30773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:32.751569 30773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:32.751652 30773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:32.751736 30773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:32.756886 30773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:32.757886 30773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:32.758893 30773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:32.768504 30778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:32.768520 30781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:32.768477 30779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:32.768957 30773 server_base.cc:1034] running on GCE node
I20260928 16:34:32.769381 30773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:32.770044 30773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:32.771210 30773 hybrid_clock.cc:648] HybridClock initialized: now 1790613272771170 us; error 52 us; skew 500 ppm
Sep 28 16:34:32 dist-test-slave-g9sh krb5kdc[30532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613272, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:32.773926 30773 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:34:32.774823 30773 webserver.cc:460] Webserver started at http://127.27.122.194:44085/ using document root <none> and password file <none>
I20260928 16:34:32.775409 30773 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:32.775501 30773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:32.775745 30773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:32.777796 30773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "472783a7cd0f46c4a2a2836ccfd93bff"
format_stamp: "Formatted at 2026-09-28 16:34:32 on dist-test-slave-g9sh"
server_key: "67fbf3c75c3bb340706d161806dd783d"
server_key_iv: "0fbb73d5d8bb81dff2f74b95bc98d679"
server_key_version: "encryptionkey@0"
I20260928 16:34:32.778371 30773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "472783a7cd0f46c4a2a2836ccfd93bff"
format_stamp: "Formatted at 2026-09-28 16:34:32 on dist-test-slave-g9sh"
server_key: "67fbf3c75c3bb340706d161806dd783d"
server_key_iv: "0fbb73d5d8bb81dff2f74b95bc98d679"
server_key_version: "encryptionkey@0"
I20260928 16:34:32.782065 30773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 16:34:32.784428 30788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:32.785686 30773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:34:32.785833 30773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "472783a7cd0f46c4a2a2836ccfd93bff"
format_stamp: "Formatted at 2026-09-28 16:34:32 on dist-test-slave-g9sh"
server_key: "67fbf3c75c3bb340706d161806dd783d"
server_key_iv: "0fbb73d5d8bb81dff2f74b95bc98d679"
server_key_version: "encryptionkey@0"
I20260928 16:34:32.785949 30773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:32.795439 30773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:32.798354 30773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:32.798550 30773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:32.799144 30773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:32.799968 30773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:32.800040 30773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:32.800107 30773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:32.800158 30773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:32.809132 30773 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:43929
I20260928 16:34:32.809168 30901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:43929 every 8 connection(s)
I20260928 16:34:32.810027 30773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:32.813628 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 30773
I20260928 16:34:32.813746 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:34:32.814046 28139 external_mini_cluster.cc:1428] Setting key 4dd1d9ed7611996a5a473c322cf75217
Sep 28 16:34:32 dist-test-slave-g9sh krb5kdc[30532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613272, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
I20260928 16:34:32.825169 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:32.811804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:43789 (local address 127.27.122.254:43551)
0928 16:34:32.811939 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:32.811943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:32.813214 (+ 1271us) server_negotiation.cc:407] Connection header received
0928 16:34:32.814392 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:32.814395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:32.814468 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:32.814594 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:32.816300 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:32.816860 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:32.818181 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:32.818593 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:32.818717 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:32.821572 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:32.821575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:32.821577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:32.821601 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:32.822978 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:32.823613 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:32.823615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:32.823616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:32.823687 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:32.824051 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:32.824053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:32.824054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:32.824268 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:32.824346 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:32.824846 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 16:34:32.824994 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:34:32.825920 30904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:32.812168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:43551 (local address 127.27.122.194:43789)
0928 16:34:32.813067 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:32.813113 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:32.814143 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:32.814809 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:32.814819 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:32.815347 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:32.816128 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:32.816153 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:32.817025 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:32.817032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:32.818057 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:32.818067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:32.818739 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:32.818742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:32.818880 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:32.819474 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:32.819490 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:32.821403 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:32.823112 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:32.823122 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:32.823138 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:32.823509 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:32.823785 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:32.823788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:32.823790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:32.823955 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:32.824380 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:32.824386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:32.824640 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 16:34:32.824795 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 16:34:32.825111 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":527,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
I20260928 16:34:32.827088 30902 heartbeater.cc:347] Connected to a master server at 127.27.122.254:43551
I20260928 16:34:32.827361 30902 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:32.827950 30902 heartbeater.cc:511] Master 127.27.122.254:43551 requested a full tablet report, sending...
I20260928 16:34:32.829151 30577 ts_manager.cc:194] Registered new tserver with Master: 472783a7cd0f46c4a2a2836ccfd93bff (127.27.122.194:43929)
I20260928 16:34:32.829726 30577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:43789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:34:32.844048 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:43551
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:32.953644 30907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:32.953882 30907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:32.953972 30907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:32.957944 30907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:32.957994 30907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:32.958050 30907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:32.963054 30907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:43551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:32.963990 30907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:32.964963 30907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:32.974537 30913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:32.974537 30916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:32.974593 30907 server_base.cc:1034] running on GCE node
W20260928 16:34:32.974534 30914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:32.975239 30907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:32.975921 30907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:32.977159 30907 hybrid_clock.cc:648] HybridClock initialized: now 1790613272977123 us; error 45 us; skew 500 ppm
Sep 28 16:34:32 dist-test-slave-g9sh krb5kdc[30532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613272, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:32.979832 30907 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:34:32.980749 30907 webserver.cc:460] Webserver started at http://127.27.122.195:42675/ using document root <none> and password file <none>
I20260928 16:34:32.981369 30907 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:32.981462 30907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:32.981706 30907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:32.983737 30907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "84a18e87e27241b98603a8c24f39175d"
format_stamp: "Formatted at 2026-09-28 16:34:32 on dist-test-slave-g9sh"
server_key: "c5f6657dca1867c01d74dcf90933bf86"
server_key_iv: "ffc04ab67def2c0ac7a2bc77cc0a857d"
server_key_version: "encryptionkey@0"
I20260928 16:34:32.984300 30907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "84a18e87e27241b98603a8c24f39175d"
format_stamp: "Formatted at 2026-09-28 16:34:32 on dist-test-slave-g9sh"
server_key: "c5f6657dca1867c01d74dcf90933bf86"
server_key_iv: "ffc04ab67def2c0ac7a2bc77cc0a857d"
server_key_version: "encryptionkey@0"
I20260928 16:34:32.987994 30907 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:34:32.990540 30923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:32.991686 30907 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:34:32.991823 30907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "84a18e87e27241b98603a8c24f39175d"
format_stamp: "Formatted at 2026-09-28 16:34:32 on dist-test-slave-g9sh"
server_key: "c5f6657dca1867c01d74dcf90933bf86"
server_key_iv: "ffc04ab67def2c0ac7a2bc77cc0a857d"
server_key_version: "encryptionkey@0"
I20260928 16:34:32.991935 30907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:33.005685 30907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:33.008375 30907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:33.008579 30907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:33.009188 30907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:33.010078 30907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:33.010152 30907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:33.010224 30907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:33.010267 30907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:33.019402 30907 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:34727
I20260928 16:34:33.019433 31036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:34727 every 8 connection(s)
I20260928 16:34:33.020310 30907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
Sep 28 16:34:33 dist-test-slave-g9sh krb5kdc[30532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613272, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:33.030190 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 30907
I20260928 16:34:33.030329 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:34:33.030629 28139 external_mini_cluster.cc:1428] Setting key efdc4f57e0324dea375ef6d3231995ac
I20260928 16:34:33.034267 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:33.022114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:41975 (local address 127.27.122.254:43551)
0928 16:34:33.022248 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:33.022251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:33.023091 (+ 840us) server_negotiation.cc:407] Connection header received
0928 16:34:33.024403 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:33.024407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:33.024466 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:33.024569 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:33.025791 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:33.026329 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:33.027499 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:33.027737 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:33.027855 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:33.030463 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:33.030466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:33.030468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:33.030493 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:33.032205 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:33.032710 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:33.032716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:33.032717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:33.032804 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:33.033196 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:33.033199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:33.033201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:33.033386 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:33.033493 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:33.033975 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 16:34:33.034104 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:34:33.035000 31039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:33.022502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:43551 (local address 127.27.122.195:41975)
0928 16:34:33.022942 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:33.022983 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:33.024181 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:33.024749 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:33.024756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:33.025176 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:33.025639 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:33.025653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:33.026461 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:33.026465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:33.027364 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:33.027373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:33.027892 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:33.027896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:33.028095 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:33.028620 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:33.028635 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:33.030251 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:33.032320 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:33.032326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:33.032339 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:33.032603 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:33.032932 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:33.032935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:33.032937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:33.033088 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:33.033493 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:33.033500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:33.033739 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 16:34:33.033916 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 16:34:33.034236 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260928 16:34:33.036044 31037 heartbeater.cc:347] Connected to a master server at 127.27.122.254:43551
I20260928 16:34:33.036288 31037 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:33.036746 31037 heartbeater.cc:511] Master 127.27.122.254:43551 requested a full tablet report, sending...
I20260928 16:34:33.037768 30577 ts_manager.cc:194] Registered new tserver with Master: 84a18e87e27241b98603a8c24f39175d (127.27.122.195:34727)
I20260928 16:34:33.038326 30577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:41975
I20260928 16:34:33.044504 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:34:33.057840 30616 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:34:33.056471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58672 (local address 127.27.122.254:43551)
0928 16:34:33.056612 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:33.056616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:33.056796 (+ 180us) server_negotiation.cc:407] Connection header received
0928 16:34:33.056938 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:33.056943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:33.056994 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:33.057067 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:33.057737 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58672: BlockingRecv error: recv got EOF from 127.0.0.1:58672 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:34:33.058784 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 30639
I20260928 16:34:33.064338 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 30773
I20260928 16:34:33.070036 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 30907
I20260928 16:34:33.075237 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 30545
2026-09-28T16:34:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3866 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:33 dist-test-slave-g9sh krb5kdc[31049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:33 dist-test-slave-g9sh krb5kdc[31049](info): set up 2 sockets
Sep 28 16:34:33 dist-test-slave-g9sh krb5kdc[31049](info): commencing operation
krb5kdc: starting...
W20260928 16:34:34.115509 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:34 dist-test-slave-g9sh krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:34:34.239648 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:42141
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:42141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:34.349984 31062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:34.350211 31062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:34.350248 31062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:34.354164 31062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:34.354240 31062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:34.354272 31062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:34.354308 31062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:34.354342 31062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:34.359308 31062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:42141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:42141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:34.360201 31062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:34.361123 31062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:34.369552 31070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:34.369601 31068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:34.369688 31067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:34.369598 31062 server_base.cc:1034] running on GCE node
I20260928 16:34:34.370378 31062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:34.371482 31062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:34.372702 31062 hybrid_clock.cc:648] HybridClock initialized: now 1790613274372683 us; error 33 us; skew 500 ppm
Sep 28 16:34:34 dist-test-slave-g9sh krb5kdc[31049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613274, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:34.375432 31062 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:34:34.376299 31062 webserver.cc:460] Webserver started at http://127.27.122.254:46857/ using document root <none> and password file <none>
I20260928 16:34:34.376863 31062 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:34.376988 31062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:34.377168 31062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:34.379123 31062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "e715eaa05bf94641964e87b9b9a88eaf"
format_stamp: "Formatted at 2026-09-28 16:34:34 on dist-test-slave-g9sh"
server_key: "20868ef25ac0808fe94ab9c5165e52a4"
server_key_iv: "6cc9568ab90192550a7418a5aa5ebc75"
server_key_version: "encryptionkey@0"
I20260928 16:34:34.379634 31062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "e715eaa05bf94641964e87b9b9a88eaf"
format_stamp: "Formatted at 2026-09-28 16:34:34 on dist-test-slave-g9sh"
server_key: "20868ef25ac0808fe94ab9c5165e52a4"
server_key_iv: "6cc9568ab90192550a7418a5aa5ebc75"
server_key_version: "encryptionkey@0"
I20260928 16:34:34.383157 31062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 16:34:34.385470 31077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:34.386632 31062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:34.386770 31062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "e715eaa05bf94641964e87b9b9a88eaf"
format_stamp: "Formatted at 2026-09-28 16:34:34 on dist-test-slave-g9sh"
server_key: "20868ef25ac0808fe94ab9c5165e52a4"
server_key_iv: "6cc9568ab90192550a7418a5aa5ebc75"
server_key_version: "encryptionkey@0"
I20260928 16:34:34.386876 31062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:34.397467 31062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:34.400193 31062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:34.400384 31062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:34.407291 31062 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:42141
I20260928 16:34:34.407552 31129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:42141 every 8 connection(s)
I20260928 16:34:34.408264 31062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:34.410883 31130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:34.415284 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 31062
I20260928 16:34:34.415385 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:34.415669 28139 external_mini_cluster.cc:1428] Setting key 0aaca4d870eaaaa5c36093ef3c74788e
I20260928 16:34:34.416571 31130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf: Bootstrap starting.
I20260928 16:34:34.419237 31130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:34.420091 31130 log.cc:824] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:34.422047 31130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf: No bootstrap required, opened a new log
Sep 28 16:34:34 dist-test-slave-g9sh krb5kdc[31049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:34.426034 31130 raft_consensus.cc:348] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e715eaa05bf94641964e87b9b9a88eaf" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42141 } }
I20260928 16:34:34.426275 31130 raft_consensus.cc:374] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:34.426350 31130 raft_consensus.cc:729] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e715eaa05bf94641964e87b9b9a88eaf, State: Initialized, Role: FOLLOWER
I20260928 16:34:34.426813 31130 consensus_queue.cc:260] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e715eaa05bf94641964e87b9b9a88eaf" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42141 } }
I20260928 16:34:34.426931 31130 raft_consensus.cc:388] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:34.427009 31130 raft_consensus.cc:482] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:34.427115 31130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:34.428007 31130 raft_consensus.cc:504] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e715eaa05bf94641964e87b9b9a88eaf" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42141 } }
I20260928 16:34:34.428356 31130 leader_election.cc:302] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e715eaa05bf94641964e87b9b9a88eaf; no voters:
I20260928 16:34:34.428704 31130 leader_election.cc:288] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:34.428802 31135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:34.429026 31135 raft_consensus.cc:686] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [term 1 LEADER]: Becoming Leader. State: Replica: e715eaa05bf94641964e87b9b9a88eaf, State: Running, Role: LEADER
I20260928 16:34:34.429409 31135 consensus_queue.cc:237] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e715eaa05bf94641964e87b9b9a88eaf" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42141 } }
I20260928 16:34:34.430115 31130 sys_catalog.cc:564] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:34.431411 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:34.417046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39992 (local address 127.27.122.254:42141)
0928 16:34:34.417467 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:34.417489 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:34.417529 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:34:34.418373 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:34.418396 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:34.418808 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:34.419172 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:34.419921 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:34.421286 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:34.422149 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:34.422826 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:34.422980 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:34.425716 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:34.425721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:34.425734 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:34.425759 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:34.427699 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:34.428210 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:34.428215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:34.428221 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:34.428346 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:34.428636 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:34.428639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:34.428640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:34.429271 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:34.429477 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:34.430086 (+ 609us) server_negotiation.cc:299] Negotiation successful
0928 16:34:34.430382 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260928 16:34:34.432252 31137 sys_catalog.cc:455] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e715eaa05bf94641964e87b9b9a88eaf. Latest consensus state: current_term: 1 leader_uuid: "e715eaa05bf94641964e87b9b9a88eaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e715eaa05bf94641964e87b9b9a88eaf" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42141 } } }
I20260928 16:34:34.432374 31137 sys_catalog.cc:458] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:34.432693 31136 sys_catalog.cc:455] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e715eaa05bf94641964e87b9b9a88eaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e715eaa05bf94641964e87b9b9a88eaf" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42141 } } }
I20260928 16:34:34.433557 31136 sys_catalog.cc:458] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:34.434775 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 16:34:34.435191 31150 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:34:34.435241 31150 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:34:34.435320 31151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:34.435983 31151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:34.441707 31151 catalog_manager.cc:1347] Generated new cluster ID: 5c28e380641e446bb51968010569cb0e
I20260928 16:34:34.441766 31151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:34.459957 31133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:34.454404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.27.122.254:42141)
0928 16:34:34.454517 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:34.454521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:34.454709 (+ 188us) server_negotiation.cc:407] Connection header received
0928 16:34:34.454824 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:34.454828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:34.454879 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:34.454975 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:34.455527 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:34.456348 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:34.457491 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:34.457910 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:34.458051 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:34.458908 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:34.458912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:34.458915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:34.459501 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:34.459535 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:34.459562 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:34.459655 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:34.459753 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 16:34:34.459814 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 16:34:34.469038 31151 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:34.470227 31151 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:34.480492 31151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e715eaa05bf94641964e87b9b9a88eaf: Generated new TSK 0
I20260928 16:34:34.481307 31151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:34:34.487962 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 31062
2026-09-28T16:34:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1411 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:34 dist-test-slave-g9sh krb5kdc[31163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:34 dist-test-slave-g9sh krb5kdc[31163](info): set up 2 sockets
Sep 28 16:34:34 dist-test-slave-g9sh krb5kdc[31163](info): commencing operation
krb5kdc: starting...
W20260928 16:34:35.543951 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:35 dist-test-slave-g9sh krb5kdc[31163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:34:35.663618 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:34709
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:44457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:34709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:35.772866 31176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:35.773125 31176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:35.773165 31176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:35.777076 31176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:35.777125 31176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:35.777143 31176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:35.777159 31176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:35.777175 31176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:35.782338 31176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:34709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:34709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:35.783259 31176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:35.784189 31176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:35.792706 31182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:35.792711 31184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:35.792757 31181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:35.792863 31176 server_base.cc:1034] running on GCE node
I20260928 16:34:35.793517 31176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:35.794548 31176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:35.795712 31176 hybrid_clock.cc:648] HybridClock initialized: now 1790613275795687 us; error 42 us; skew 500 ppm
Sep 28 16:34:35 dist-test-slave-g9sh krb5kdc[31163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613275, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:35.798573 31176 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:34:35.799497 31176 webserver.cc:460] Webserver started at http://127.27.122.254:39891/ using document root <none> and password file <none>
I20260928 16:34:35.800060 31176 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:35.800151 31176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:35.800369 31176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:35.802444 31176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "47d7f4e734e44b48b8fa980b9eeb3bf9"
format_stamp: "Formatted at 2026-09-28 16:34:35 on dist-test-slave-g9sh"
server_key: "aa1d0d3eaebc5dc7e5170b28eef92c2a"
server_key_iv: "ef39b7c2d4dd5b14b0dc809fb87df74f"
server_key_version: "encryptionkey@0"
I20260928 16:34:35.803033 31176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "47d7f4e734e44b48b8fa980b9eeb3bf9"
format_stamp: "Formatted at 2026-09-28 16:34:35 on dist-test-slave-g9sh"
server_key: "aa1d0d3eaebc5dc7e5170b28eef92c2a"
server_key_iv: "ef39b7c2d4dd5b14b0dc809fb87df74f"
server_key_version: "encryptionkey@0"
I20260928 16:34:35.806815 31176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 16:34:35.809290 31191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:35.810459 31176 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:35.810611 31176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "47d7f4e734e44b48b8fa980b9eeb3bf9"
format_stamp: "Formatted at 2026-09-28 16:34:35 on dist-test-slave-g9sh"
server_key: "aa1d0d3eaebc5dc7e5170b28eef92c2a"
server_key_iv: "ef39b7c2d4dd5b14b0dc809fb87df74f"
server_key_version: "encryptionkey@0"
I20260928 16:34:35.810724 31176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:35.828704 31176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:35.832109 31176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:35.832322 31176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:35.839725 31176 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:34709
I20260928 16:34:35.839999 31243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:34709 every 8 connection(s)
I20260928 16:34:35.840731 31176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:35.843570 31244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:35.848948 31244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9: Bootstrap starting.
I20260928 16:34:35.849447 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 31176
I20260928 16:34:35.849579 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:35.849857 28139 external_mini_cluster.cc:1428] Setting key 80372714849677edcf3d2102c4d30600
I20260928 16:34:35.851640 31244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:35.852375 31244 log.cc:824] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:35.854521 31244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9: No bootstrap required, opened a new log
Sep 28 16:34:35 dist-test-slave-g9sh krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:35.859011 31244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7f4e734e44b48b8fa980b9eeb3bf9" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34709 } }
I20260928 16:34:35.859265 31244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:35.859370 31244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d7f4e734e44b48b8fa980b9eeb3bf9, State: Initialized, Role: FOLLOWER
I20260928 16:34:35.859853 31244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7f4e734e44b48b8fa980b9eeb3bf9" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34709 } }
I20260928 16:34:35.860014 31244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:35.860090 31244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:35.860203 31244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:35.861478 31244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7f4e734e44b48b8fa980b9eeb3bf9" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34709 } }
I20260928 16:34:35.861879 31244 leader_election.cc:302] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47d7f4e734e44b48b8fa980b9eeb3bf9; no voters:
I20260928 16:34:35.862233 31244 leader_election.cc:288] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:35.862341 31249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:35.862601 31249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [term 1 LEADER]: Becoming Leader. State: Replica: 47d7f4e734e44b48b8fa980b9eeb3bf9, State: Running, Role: LEADER
I20260928 16:34:35.862991 31249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7f4e734e44b48b8fa980b9eeb3bf9" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34709 } }
I20260928 16:34:35.863541 31244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:35.864228 31251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47d7f4e734e44b48b8fa980b9eeb3bf9. Latest consensus state: current_term: 1 leader_uuid: "47d7f4e734e44b48b8fa980b9eeb3bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7f4e734e44b48b8fa980b9eeb3bf9" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34709 } } }
I20260928 16:34:35.864952 31251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:35.865269 31250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47d7f4e734e44b48b8fa980b9eeb3bf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47d7f4e734e44b48b8fa980b9eeb3bf9" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34709 } } }
I20260928 16:34:35.865433 31250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:35.866024 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:35.851135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.27.122.254:34709)
0928 16:34:35.851550 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:35.851566 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:35.851602 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:34:35.852268 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:35.852285 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:35.852663 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:35.853078 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:35.853912 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:35.854959 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:35.856203 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:35.856683 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:35.856804 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:35.859913 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:35.859919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:35.859937 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:35.859971 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:35.862174 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:35.862745 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:35.862751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:35.862757 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:35.862910 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:35.863226 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:35.863229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:35.863231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:35.863742 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:35.863976 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:35.864685 (+ 709us) server_negotiation.cc:299] Negotiation successful
0928 16:34:35.865029 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":140,"threads_started":1}
W20260928 16:34:35.868546 31264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:34:35.868636 31264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:34:35.869071 31258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:35.869768 31258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:35.874908 31258 catalog_manager.cc:1347] Generated new cluster ID: de54878cb0474b779efc875647582793
I20260928 16:34:35.874977 31258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:35.887249 31258 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:35.888185 31258 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:35.896106 31258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47d7f4e734e44b48b8fa980b9eeb3bf9: Generated new TSK 0
I20260928 16:34:35.896754 31258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:34:35.928371 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34709
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:44457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:36.053347 31270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:36.053578 31270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:36.053686 31270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:36.057643 31270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:36.057731 31270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:36.057862 31270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:36.062999 31270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:36.064186 31270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:36.065325 31270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:36.075073 31278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:36.075071 31275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:36.075114 31276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:36.075338 31270 server_base.cc:1034] running on GCE node
I20260928 16:34:36.075933 31270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:36.076534 31270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:36.077702 31270 hybrid_clock.cc:648] HybridClock initialized: now 1790613276077677 us; error 34 us; skew 500 ppm
Sep 28 16:34:36 dist-test-slave-g9sh krb5kdc[31163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613276, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:36.080562 31270 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:34:36.081475 31270 webserver.cc:460] Webserver started at http://127.27.122.193:43179/ using document root <none> and password file <none>
I20260928 16:34:36.082010 31270 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:36.082099 31270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:36.082325 31270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:36.084332 31270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "87d1455d7b2949469e05670197e57753"
format_stamp: "Formatted at 2026-09-28 16:34:36 on dist-test-slave-g9sh"
server_key: "6aa9caaae77a18e1da8af778defb0637"
server_key_iv: "d3b8fdc7498a56d84db55d4047375c24"
server_key_version: "encryptionkey@0"
I20260928 16:34:36.084882 31270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "87d1455d7b2949469e05670197e57753"
format_stamp: "Formatted at 2026-09-28 16:34:36 on dist-test-slave-g9sh"
server_key: "6aa9caaae77a18e1da8af778defb0637"
server_key_iv: "d3b8fdc7498a56d84db55d4047375c24"
server_key_version: "encryptionkey@0"
I20260928 16:34:36.088402 31270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:34:36.090793 31285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:36.091917 31270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:36.092048 31270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "87d1455d7b2949469e05670197e57753"
format_stamp: "Formatted at 2026-09-28 16:34:36 on dist-test-slave-g9sh"
server_key: "6aa9caaae77a18e1da8af778defb0637"
server_key_iv: "d3b8fdc7498a56d84db55d4047375c24"
server_key_version: "encryptionkey@0"
I20260928 16:34:36.092156 31270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:36.111711 31270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:36.114792 31270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:36.115020 31270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:36.115624 31270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:36.116475 31270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:36.116547 31270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:36.116613 31270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:36.116662 31270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:36.125442 31270 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:36997
I20260928 16:34:36.125465 31398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:36997 every 8 connection(s)
I20260928 16:34:36.126281 31270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:36.134989 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 31270
I20260928 16:34:36.135154 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:34:36.135468 28139 external_mini_cluster.cc:1428] Setting key 4083e080cd5032cbf0a0dd52f4d12c1d
Sep 28 16:34:36 dist-test-slave-g9sh krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613276, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:36.140512 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:36.128014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:53441 (local address 127.27.122.254:34709)
0928 16:34:36.128171 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:36.128176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:36.129119 (+ 943us) server_negotiation.cc:407] Connection header received
0928 16:34:36.130480 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:36.130483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:36.130532 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:36.130607 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:36.131772 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.132304 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.133483 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.133717 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.133796 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:36.136270 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:36.136277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:36.136280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:36.136313 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:36.138155 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:36.138644 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:36.138651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:36.138655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:36.138759 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:36.139204 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:36.139206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:36.139207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:36.139376 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:36.139454 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:36.140122 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 16:34:36.140268 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:34:36.141285 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:36.128438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:34709 (local address 127.27.122.193:53441)
0928 16:34:36.128960 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:36.129019 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:36.130288 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:36.130811 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:36.130819 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:36.131175 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:36.131643 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:36.131656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.132412 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.132414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:36.133379 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:36.133389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.133849 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.133851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:36.133984 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:36.134518 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:36.134534 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:36.136103 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:36.138276 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:36.138281 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:36.138293 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:36.138537 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:36.138879 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:36.138886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:36.138889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:36.139085 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:36.139482 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:36.139491 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:36.139826 (+ 335us) client_negotiation.cc:769] Sending connection context
0928 16:34:36.140048 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 16:34:36.140360 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":122,"threads_started":1}
I20260928 16:34:36.142685 31399 heartbeater.cc:347] Connected to a master server at 127.27.122.254:34709
I20260928 16:34:36.142927 31399 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:36.143409 31399 heartbeater.cc:511] Master 127.27.122.254:34709 requested a full tablet report, sending...
I20260928 16:34:36.144954 31208 ts_manager.cc:194] Registered new tserver with Master: 87d1455d7b2949469e05670197e57753 (127.27.122.193:36997)
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:34:36.146129 31208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:53441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:34:36.162187 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34709
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:44457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:36.271554 31404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:36.271773 31404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:36.271812 31404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:36.275651 31404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:36.275697 31404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:36.275750 31404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:36.280689 31404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:36.281625 31404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:36.282596 31404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:36.291917 31410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:36.291920 31409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:36.291929 31412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:36.292443 31404 server_base.cc:1034] running on GCE node
I20260928 16:34:36.292793 31404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:36.293473 31404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:36.294675 31404 hybrid_clock.cc:648] HybridClock initialized: now 1790613276294637 us; error 48 us; skew 500 ppm
Sep 28 16:34:36 dist-test-slave-g9sh krb5kdc[31163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613276, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:36.297936 31404 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:34:36.298897 31404 webserver.cc:460] Webserver started at http://127.27.122.194:45783/ using document root <none> and password file <none>
I20260928 16:34:36.299542 31404 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:36.299647 31404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:36.299902 31404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:36.302271 31404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "7ddbe36eefbd4d7fa05d49d1115ba1d4"
format_stamp: "Formatted at 2026-09-28 16:34:36 on dist-test-slave-g9sh"
server_key: "271423317a82fbb49811c8c30b7bd8cc"
server_key_iv: "08c698e93facebacf10b168e10b267ee"
server_key_version: "encryptionkey@0"
I20260928 16:34:36.302901 31404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "7ddbe36eefbd4d7fa05d49d1115ba1d4"
format_stamp: "Formatted at 2026-09-28 16:34:36 on dist-test-slave-g9sh"
server_key: "271423317a82fbb49811c8c30b7bd8cc"
server_key_iv: "08c698e93facebacf10b168e10b267ee"
server_key_version: "encryptionkey@0"
I20260928 16:34:36.307062 31404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:34:36.309746 31419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:36.311029 31404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:36.311174 31404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "7ddbe36eefbd4d7fa05d49d1115ba1d4"
format_stamp: "Formatted at 2026-09-28 16:34:36 on dist-test-slave-g9sh"
server_key: "271423317a82fbb49811c8c30b7bd8cc"
server_key_iv: "08c698e93facebacf10b168e10b267ee"
server_key_version: "encryptionkey@0"
I20260928 16:34:36.311295 31404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:36.379537 31404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:36.382877 31404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:36.383104 31404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:36.383765 31404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:36.384665 31404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:36.384742 31404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:36.384822 31404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:36.384881 31404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:36.394707 31404 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:34381
I20260928 16:34:36.394742 31532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:34381 every 8 connection(s)
I20260928 16:34:36.395757 31404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:36.398932 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 31404
I20260928 16:34:36.399056 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:34:36.399441 28139 external_mini_cluster.cc:1428] Setting key 0d3e091b50a8d19eb23be2e92151f2e6
Sep 28 16:34:36 dist-test-slave-g9sh krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613276, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
I20260928 16:34:36.411868 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:36.397789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:37983 (local address 127.27.122.254:34709)
0928 16:34:36.397927 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:36.397931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:36.398763 (+ 832us) server_negotiation.cc:407] Connection header received
0928 16:34:36.400636 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:36.400639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:36.400688 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:36.400776 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:36.402600 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.403355 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.404613 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.405060 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.405187 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:36.407908 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:36.407913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:36.407916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:36.407951 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:36.409698 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:36.410195 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:36.410197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:36.410198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:36.410273 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:36.410726 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:36.410730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:36.410731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:36.410917 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:36.411005 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:36.411643 (+ 638us) server_negotiation.cc:299] Negotiation successful
0928 16:34:36.411746 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
I20260928 16:34:36.413059 31535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:36.398109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:34709 (local address 127.27.122.194:37983)
0928 16:34:36.398566 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:36.398613 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:36.400419 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:36.401086 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:36.401096 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:36.401658 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:36.402424 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:36.402451 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.403493 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.403496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:36.404469 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:36.404479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.405237 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.405242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:36.405458 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:36.406199 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:36.406216 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:36.407730 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:36.409835 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:36.409839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:36.409850 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:36.410101 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:36.410353 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:36.410355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:36.410357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:36.410561 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:36.411046 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:36.411052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:36.411302 (+ 250us) client_negotiation.cc:769] Sending connection context
0928 16:34:36.411617 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 16:34:36.411898 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260928 16:34:36.414510 31533 heartbeater.cc:347] Connected to a master server at 127.27.122.254:34709
I20260928 16:34:36.414772 31533 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:36.415421 31533 heartbeater.cc:511] Master 127.27.122.254:34709 requested a full tablet report, sending...
I20260928 16:34:36.416739 31208 ts_manager.cc:194] Registered new tserver with Master: 7ddbe36eefbd4d7fa05d49d1115ba1d4 (127.27.122.194:34381)
I20260928 16:34:36.417662 31208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:37983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:34:36.428488 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34709
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:44457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:36.537626 31538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:36.537900 31538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:36.537982 31538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:36.541970 31538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:36.542047 31538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:36.542130 31538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:36.547297 31538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:36.548310 31538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:36.549329 31538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:36.558735 31544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:36.558748 31546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:36.558724 31543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:36.558887 31538 server_base.cc:1034] running on GCE node
I20260928 16:34:36.559270 31538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:36.559854 31538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:36.560974 31538 hybrid_clock.cc:648] HybridClock initialized: now 1790613276560958 us; error 26 us; skew 500 ppm
Sep 28 16:34:36 dist-test-slave-g9sh krb5kdc[31163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613276, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:36.563562 31538 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:34:36.564404 31538 webserver.cc:460] Webserver started at http://127.27.122.195:45315/ using document root <none> and password file <none>
I20260928 16:34:36.565038 31538 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:36.565104 31538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:36.565348 31538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:36.567298 31538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "d52e67d797434f73ad59609d98e541d1"
format_stamp: "Formatted at 2026-09-28 16:34:36 on dist-test-slave-g9sh"
server_key: "6a28bc0b501f1e3d3ccac9787ee0e326"
server_key_iv: "52c32575835b135375744c471dd2490e"
server_key_version: "encryptionkey@0"
I20260928 16:34:36.567847 31538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "d52e67d797434f73ad59609d98e541d1"
format_stamp: "Formatted at 2026-09-28 16:34:36 on dist-test-slave-g9sh"
server_key: "6a28bc0b501f1e3d3ccac9787ee0e326"
server_key_iv: "52c32575835b135375744c471dd2490e"
server_key_version: "encryptionkey@0"
I20260928 16:34:36.571355 31538 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 16:34:36.573678 31553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:36.574888 31538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:36.575006 31538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "d52e67d797434f73ad59609d98e541d1"
format_stamp: "Formatted at 2026-09-28 16:34:36 on dist-test-slave-g9sh"
server_key: "6a28bc0b501f1e3d3ccac9787ee0e326"
server_key_iv: "52c32575835b135375744c471dd2490e"
server_key_version: "encryptionkey@0"
I20260928 16:34:36.575109 31538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:36.598394 31538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:36.601606 31538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:36.601752 31538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:36.602294 31538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:36.603052 31538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:36.603101 31538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:36.603140 31538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:36.603163 31538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:36.611768 31538 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:41233
I20260928 16:34:36.611917 31666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:41233 every 8 connection(s)
I20260928 16:34:36.612809 31538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:34:36.614243 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 31538
I20260928 16:34:36.614369 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:34:36.614735 28139 external_mini_cluster.cc:1428] Setting key 400296217a35341716e0e35254cac90c
Sep 28 16:34:36 dist-test-slave-g9sh krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613276, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:36.626089 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:36.615131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:34575 (local address 127.27.122.254:34709)
0928 16:34:36.615282 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:36.615287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:36.616024 (+ 737us) server_negotiation.cc:407] Connection header received
0928 16:34:36.617591 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:36.617593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:36.617638 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:36.617706 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:36.618878 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.619427 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.620504 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.620720 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.620796 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:36.623097 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:36.623100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:36.623102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:36.623127 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:36.624306 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:36.624766 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:36.624768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:36.624769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:36.624842 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:36.625190 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:36.625193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:36.625194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:36.625356 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:36.625432 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:36.625871 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 16:34:36.625966 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:34:36.626811 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:36.615293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:34709 (local address 127.27.122.195:34575)
0928 16:34:36.615858 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:36.615914 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:36.617410 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:36.617862 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:36.617869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:36.618203 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:36.618659 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:36.618673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.619536 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.619538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:36.620399 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:36.620408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.620843 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.620846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:36.621048 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:36.621539 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:36.621555 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:36.622935 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:36.624410 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:36.624415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:36.624425 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:36.624682 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:36.624973 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:36.624975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:36.624977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:36.625109 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:36.625446 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:36.625451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:36.625670 (+ 219us) client_negotiation.cc:769] Sending connection context
0928 16:34:36.625810 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 16:34:36.626034 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":153,"threads_started":1}
I20260928 16:34:36.627934 31667 heartbeater.cc:347] Connected to a master server at 127.27.122.254:34709
I20260928 16:34:36.628167 31667 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:36.628737 31667 heartbeater.cc:511] Master 127.27.122.254:34709 requested a full tablet report, sending...
I20260928 16:34:36.629709 31208 ts_manager.cc:194] Registered new tserver with Master: d52e67d797434f73ad59609d98e541d1 (127.27.122.195:41233)
I20260928 16:34:36.630267 31208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:34575
I20260928 16:34:36.640226 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:34:36 dist-test-slave-g9sh krb5kdc[31163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613276, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 16:34:36 dist-test-slave-g9sh krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613276, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:36.664867 31247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:36.656371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.27.122.254:34709)
0928 16:34:36.656525 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:36.656531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:36.656714 (+ 183us) server_negotiation.cc:407] Connection header received
0928 16:34:36.656892 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:36.656894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:36.656987 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:36.657081 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:36.657783 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.658302 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.659185 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:36.659402 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:36.659511 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:36.661888 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:36.661892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:36.661894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:36.661925 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:36.663154 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:36.663572 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:36.663578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:36.663600 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:36.663714 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:36.663990 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:36.663995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:36.663996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:36.664221 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:36.664365 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:36.664597 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 16:34:36.664703 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 16:34:36.665377 31208 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40254
I20260928 16:34:36.666209 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 31270
I20260928 16:34:36.673532 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 31404
I20260928 16:34:36.679047 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 31538
I20260928 16:34:36.684237 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 31176
2026-09-28T16:34:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2197 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:36 dist-test-slave-g9sh krb5kdc[31679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:36 dist-test-slave-g9sh krb5kdc[31679](info): set up 2 sockets
Sep 28 16:34:36 dist-test-slave-g9sh krb5kdc[31679](info): commencing operation
krb5kdc: starting...
W20260928 16:34:37.727499 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:37 dist-test-slave-g9sh krb5kdc[31679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:34:37.843926 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:36541
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:33653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:36541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:37.951520 31692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:37.951783 31692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:37.951854 31692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:37.955895 31692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:37.955973 31692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:37.956012 31692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:37.956075 31692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:37.956113 31692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:37.961135 31692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:33653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:36541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:36541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:37.962091 31692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:37.963057 31692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:37.971572 31700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:37.971776 31692 server_base.cc:1034] running on GCE node
W20260928 16:34:37.971606 31697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:37.971585 31698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:37.972497 31692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:37.973568 31692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:37.974789 31692 hybrid_clock.cc:648] HybridClock initialized: now 1790613277974776 us; error 24 us; skew 500 ppm
Sep 28 16:34:37 dist-test-slave-g9sh krb5kdc[31679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613277, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:37.985410 31692 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:34:37.986304 31692 webserver.cc:460] Webserver started at http://127.27.122.254:42901/ using document root <none> and password file <none>
I20260928 16:34:37.986864 31692 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:37.986956 31692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:37.987195 31692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:37.989326 31692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "ca8a288cd7424d7ca81ac848263b4601"
format_stamp: "Formatted at 2026-09-28 16:34:37 on dist-test-slave-g9sh"
server_key: "c0c904ae2426ea4e6586d101ed0193fa"
server_key_iv: "bc6541a5e1d3478e406646e5d0c1bf5e"
server_key_version: "encryptionkey@0"
I20260928 16:34:37.989894 31692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "ca8a288cd7424d7ca81ac848263b4601"
format_stamp: "Formatted at 2026-09-28 16:34:37 on dist-test-slave-g9sh"
server_key: "c0c904ae2426ea4e6586d101ed0193fa"
server_key_iv: "bc6541a5e1d3478e406646e5d0c1bf5e"
server_key_version: "encryptionkey@0"
I20260928 16:34:37.993602 31692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:34:37.995900 31707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:37.997184 31692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 16:34:37.997321 31692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "ca8a288cd7424d7ca81ac848263b4601"
format_stamp: "Formatted at 2026-09-28 16:34:37 on dist-test-slave-g9sh"
server_key: "c0c904ae2426ea4e6586d101ed0193fa"
server_key_iv: "bc6541a5e1d3478e406646e5d0c1bf5e"
server_key_version: "encryptionkey@0"
I20260928 16:34:37.997428 31692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:38.009986 31692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:38.012588 31692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:38.012770 31692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:38.019572 31692 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:36541
I20260928 16:34:38.019595 31759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:36541 every 8 connection(s)
I20260928 16:34:38.020440 31692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:38.023075 31760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:38.028218 31760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601: Bootstrap starting.
I20260928 16:34:38.029165 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 31692
I20260928 16:34:38.029311 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:38.029587 28139 external_mini_cluster.cc:1428] Setting key eae32e840e0cc0644facfb2bc72bb9d0
I20260928 16:34:38.030902 31760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:38.031643 31760 log.cc:824] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:38.033839 31760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601: No bootstrap required, opened a new log
I20260928 16:34:38.038259 31760 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8a288cd7424d7ca81ac848263b4601" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36541 } }
I20260928 16:34:38.038471 31760 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:38.038529 31760 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8a288cd7424d7ca81ac848263b4601, State: Initialized, Role: FOLLOWER
Sep 28 16:34:38 dist-test-slave-g9sh krb5kdc[31679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:38.038986 31760 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8a288cd7424d7ca81ac848263b4601" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36541 } }
I20260928 16:34:38.039121 31760 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:38.039203 31760 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:38.039306 31760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:38.040174 31760 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8a288cd7424d7ca81ac848263b4601" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36541 } }
I20260928 16:34:38.040576 31760 leader_election.cc:302] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca8a288cd7424d7ca81ac848263b4601; no voters:
I20260928 16:34:38.040931 31760 leader_election.cc:288] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:38.041062 31765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:38.041368 31765 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [term 1 LEADER]: Becoming Leader. State: Replica: ca8a288cd7424d7ca81ac848263b4601, State: Running, Role: LEADER
I20260928 16:34:38.041718 31765 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8a288cd7424d7ca81ac848263b4601" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36541 } }
I20260928 16:34:38.042244 31760 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:38.044237 31767 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca8a288cd7424d7ca81ac848263b4601. Latest consensus state: current_term: 1 leader_uuid: "ca8a288cd7424d7ca81ac848263b4601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8a288cd7424d7ca81ac848263b4601" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36541 } } }
I20260928 16:34:38.044414 31767 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:38.044888 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:38.030685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46014 (local address 127.27.122.254:36541)
0928 16:34:38.031132 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:38.031147 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:38.031181 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:34:38.031863 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:38.031882 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:38.032276 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:38.032593 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:38.033542 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.034546 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.035751 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.036221 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.036332 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:38.039352 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:38.039376 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:38.039395 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:38.039429 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:38.041484 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:38.042014 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:38.042019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:38.042025 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:38.042160 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:38.042539 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:38.042544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:38.042547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:38.043067 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:38.043281 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:38.043568 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 16:34:38.043876 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":128,"threads_started":1}
I20260928 16:34:38.044807 31766 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca8a288cd7424d7ca81ac848263b4601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8a288cd7424d7ca81ac848263b4601" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36541 } } }
I20260928 16:34:38.045212 31766 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:38.045435 31774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:38.047961 31774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:38.053210 31774 catalog_manager.cc:1347] Generated new cluster ID: 5338cdaee8b549c0844aef6c6e6f38d3
I20260928 16:34:38.053269 31774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:38.066417 31774 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:38.067258 31774 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:38.075811 31774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca8a288cd7424d7ca81ac848263b4601: Generated new TSK 0
I20260928 16:34:38.076515 31774 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:34:38.116145 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36541
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:33653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:38.239524 31786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:38.239805 31786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:38.239874 31786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:38.246066 31786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:38.246161 31786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:38.246276 31786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:38.253147 31786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:33653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:38.254465 31786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:38.255518 31786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:38.265081 31794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:38.265106 31792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:38.265141 31786 server_base.cc:1034] running on GCE node
W20260928 16:34:38.265312 31791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:38.265713 31786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:38.266314 31786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:38.267506 31786 hybrid_clock.cc:648] HybridClock initialized: now 1790613278267478 us; error 67 us; skew 500 ppm
Sep 28 16:34:38 dist-test-slave-g9sh krb5kdc[31679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613278, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:38.270362 31786 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:34:38.271127 31786 webserver.cc:460] Webserver started at http://127.27.122.193:46237/ using document root <none> and password file <none>
I20260928 16:34:38.271628 31786 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:38.271685 31786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:38.271847 31786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:38.273911 31786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "1d0135bcf2a44b1e9244779636340218"
format_stamp: "Formatted at 2026-09-28 16:34:38 on dist-test-slave-g9sh"
server_key: "6dc8e1abd454270fd1942c602e6deef3"
server_key_iv: "f51131f335ed15a2ef091f14b0f59515"
server_key_version: "encryptionkey@0"
I20260928 16:34:38.274428 31786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "1d0135bcf2a44b1e9244779636340218"
format_stamp: "Formatted at 2026-09-28 16:34:38 on dist-test-slave-g9sh"
server_key: "6dc8e1abd454270fd1942c602e6deef3"
server_key_iv: "f51131f335ed15a2ef091f14b0f59515"
server_key_version: "encryptionkey@0"
I20260928 16:34:38.278036 31786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 16:34:38.280346 31801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:38.281581 31786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:38.281700 31786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "1d0135bcf2a44b1e9244779636340218"
format_stamp: "Formatted at 2026-09-28 16:34:38 on dist-test-slave-g9sh"
server_key: "6dc8e1abd454270fd1942c602e6deef3"
server_key_iv: "f51131f335ed15a2ef091f14b0f59515"
server_key_version: "encryptionkey@0"
I20260928 16:34:38.281800 31786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:38.302472 31786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:38.305751 31786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:38.305951 31786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:38.306535 31786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:38.307325 31786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:38.307370 31786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:38.307492 31786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:38.307528 31786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:38.316375 31786 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:37569
I20260928 16:34:38.316408 31916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:37569 every 8 connection(s)
I20260928 16:34:38.317303 31786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:38.323102 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 31786
I20260928 16:34:38.323244 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:34:38.323529 28139 external_mini_cluster.cc:1428] Setting key 47e2cb81fe7e0d25fbbe064a0447c4d9
Sep 28 16:34:38 dist-test-slave-g9sh krb5kdc[31679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613278, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:38.331506 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:38.319015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:59535 (local address 127.27.122.254:36541)
0928 16:34:38.319163 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:38.319167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:38.319906 (+ 739us) server_negotiation.cc:407] Connection header received
0928 16:34:38.321444 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:38.321448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:38.321506 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:38.321596 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:38.322734 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.323444 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.324639 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.324982 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.325103 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:38.327503 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:38.327509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:38.327515 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:38.327547 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:38.329254 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:38.329742 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:38.329748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:38.329753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:38.329848 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:38.330261 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:38.330267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:38.330271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:38.330539 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:38.330674 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:38.331163 (+ 489us) server_negotiation.cc:299] Negotiation successful
0928 16:34:38.331294 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:34:38.332257 31919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:38.319343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:36541 (local address 127.27.122.193:59535)
0928 16:34:38.319760 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:38.319801 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:38.321231 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:38.321761 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:38.321768 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:38.322123 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:38.322586 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:38.322599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.323574 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.323583 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:38.324494 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:38.324504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.325151 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.325155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:38.325327 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:38.325911 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:38.325928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:38.327338 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:38.329369 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:38.329374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:38.329384 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:38.329635 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:38.329982 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:38.329985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:38.329987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:38.330151 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:38.330666 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:38.330671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:38.330918 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 16:34:38.331077 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 16:34:38.331343 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:34:38.333534 31917 heartbeater.cc:347] Connected to a master server at 127.27.122.254:36541
I20260928 16:34:38.333819 31917 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:38.334307 31917 heartbeater.cc:511] Master 127.27.122.254:36541 requested a full tablet report, sending...
I20260928 16:34:38.335969 31724 ts_manager.cc:194] Registered new tserver with Master: 1d0135bcf2a44b1e9244779636340218 (127.27.122.193:37569)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
I20260928 16:34:38.337707 31724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:59535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:34:38.351581 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36541
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:33653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:38.459676 31922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:38.459904 31922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:38.459945 31922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:38.463912 31922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:38.463958 31922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:38.464043 31922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:38.469040 31922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:33653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:38.469990 31922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:38.470955 31922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:38.480465 31930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:38.480476 31928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:38.480533 31922 server_base.cc:1034] running on GCE node
W20260928 16:34:38.480443 31927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:38.481178 31922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:38.481745 31922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:38.482911 31922 hybrid_clock.cc:648] HybridClock initialized: now 1790613278482889 us; error 24 us; skew 500 ppm
Sep 28 16:34:38 dist-test-slave-g9sh krb5kdc[31679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613278, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:38.486126 31922 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:34:38.486987 31922 webserver.cc:460] Webserver started at http://127.27.122.194:46333/ using document root <none> and password file <none>
I20260928 16:34:38.487533 31922 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:38.487602 31922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:38.487838 31922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:38.489881 31922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "e3ca2624a20144f9901aa7500185a18d"
format_stamp: "Formatted at 2026-09-28 16:34:38 on dist-test-slave-g9sh"
server_key: "691f7c18b2a2d076642b25061c40201f"
server_key_iv: "450bc785383123d1d97a61c820f79d1b"
server_key_version: "encryptionkey@0"
I20260928 16:34:38.490424 31922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "e3ca2624a20144f9901aa7500185a18d"
format_stamp: "Formatted at 2026-09-28 16:34:38 on dist-test-slave-g9sh"
server_key: "691f7c18b2a2d076642b25061c40201f"
server_key_iv: "450bc785383123d1d97a61c820f79d1b"
server_key_version: "encryptionkey@0"
I20260928 16:34:38.494027 31922 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 16:34:38.496400 31937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:38.497659 31922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:34:38.497779 31922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "e3ca2624a20144f9901aa7500185a18d"
format_stamp: "Formatted at 2026-09-28 16:34:38 on dist-test-slave-g9sh"
server_key: "691f7c18b2a2d076642b25061c40201f"
server_key_iv: "450bc785383123d1d97a61c820f79d1b"
server_key_version: "encryptionkey@0"
I20260928 16:34:38.497882 31922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:38.510576 31922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:38.513411 31922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:38.513612 31922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:38.514175 31922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:38.514974 31922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:38.515020 31922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:38.515087 31922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:38.515127 31922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:38.524036 31922 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:36405
I20260928 16:34:38.524070 32051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:36405 every 8 connection(s)
I20260928 16:34:38.524945 31922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:38.526963 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 31922
I20260928 16:34:38.527104 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:34:38.527405 28139 external_mini_cluster.cc:1428] Setting key 433556329888fa5c4e010f2c366a0a35
Sep 28 16:34:38 dist-test-slave-g9sh krb5kdc[31679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613278, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
I20260928 16:34:38.540961 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:38.526688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:51761 (local address 127.27.122.254:36541)
0928 16:34:38.526861 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:38.526865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:38.527828 (+ 963us) server_negotiation.cc:407] Connection header received
0928 16:34:38.529631 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:38.529633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:38.529676 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:38.529745 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:38.531299 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.532124 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.534025 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.534239 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.534319 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:38.537164 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:38.537167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:38.537170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:38.537192 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:38.538904 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:38.539416 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:38.539420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:38.539422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:38.539506 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:38.539847 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:38.539850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:38.539852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:38.540047 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:38.540153 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:38.540646 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 16:34:38.540765 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:34:38.541749 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:38.527131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:36541 (local address 127.27.122.194:51761)
0928 16:34:38.527670 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:38.527727 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:38.529401 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:38.529967 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:38.529979 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:38.530516 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:38.531135 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:38.531150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.532257 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.532264 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:38.533904 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:38.533917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.534407 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.534414 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:38.534653 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:38.535301 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:38.535331 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:38.536999 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:38.539023 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:38.539030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:38.539045 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:38.539308 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:38.539600 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:38.539604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:38.539606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:38.539743 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:38.540153 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:38.540161 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:38.540397 (+ 236us) client_negotiation.cc:769] Sending connection context
0928 16:34:38.540589 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 16:34:38.540934 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":107,"threads_started":1}
I20260928 16:34:38.542898 32052 heartbeater.cc:347] Connected to a master server at 127.27.122.254:36541
I20260928 16:34:38.543190 32052 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:38.543751 32052 heartbeater.cc:511] Master 127.27.122.254:36541 requested a full tablet report, sending...
I20260928 16:34:38.544955 31724 ts_manager.cc:194] Registered new tserver with Master: e3ca2624a20144f9901aa7500185a18d (127.27.122.194:36405)
I20260928 16:34:38.545614 31724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:51761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:34:38.555891 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36541
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:33653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:38.666409 32057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:38.666637 32057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:38.666677 32057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:38.670683 32057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:38.670732 32057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:38.670786 32057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:38.675820 32057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:33653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:38.676733 32057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:38.677713 32057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:38.687027 32062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:38.687057 32063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:38.687222 32065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:38.687167 32057 server_base.cc:1034] running on GCE node
I20260928 16:34:38.687740 32057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:38.688316 32057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:38.689522 32057 hybrid_clock.cc:648] HybridClock initialized: now 1790613278689511 us; error 31 us; skew 500 ppm
Sep 28 16:34:38 dist-test-slave-g9sh krb5kdc[31679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613278, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:38.692461 32057 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:34:38.693389 32057 webserver.cc:460] Webserver started at http://127.27.122.195:43217/ using document root <none> and password file <none>
I20260928 16:34:38.693871 32057 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:38.693924 32057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:38.694087 32057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:38.695951 32057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "6b4fd68966fb4efd88ead93384cd43c4"
format_stamp: "Formatted at 2026-09-28 16:34:38 on dist-test-slave-g9sh"
server_key: "7be0cb9a6f663360ef09b96af80e2b4b"
server_key_iv: "75195d640461f1310a612288cf9c6ce7"
server_key_version: "encryptionkey@0"
I20260928 16:34:38.696451 32057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "6b4fd68966fb4efd88ead93384cd43c4"
format_stamp: "Formatted at 2026-09-28 16:34:38 on dist-test-slave-g9sh"
server_key: "7be0cb9a6f663360ef09b96af80e2b4b"
server_key_iv: "75195d640461f1310a612288cf9c6ce7"
server_key_version: "encryptionkey@0"
I20260928 16:34:38.699852 32057 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:34:38.702415 32072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:38.703593 32057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:34:38.703689 32057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "6b4fd68966fb4efd88ead93384cd43c4"
format_stamp: "Formatted at 2026-09-28 16:34:38 on dist-test-slave-g9sh"
server_key: "7be0cb9a6f663360ef09b96af80e2b4b"
server_key_iv: "75195d640461f1310a612288cf9c6ce7"
server_key_version: "encryptionkey@0"
I20260928 16:34:38.703768 32057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:38.732795 32057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:38.736161 32057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:38.736315 32057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:38.736846 32057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:38.737680 32057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:38.737726 32057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:38.737766 32057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:38.737789 32057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:38.747098 32057 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:44575
I20260928 16:34:38.747190 32186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:44575 every 8 connection(s)
I20260928 16:34:38.747991 32057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:34:38.751720 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 32057
I20260928 16:34:38.751852 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:34:38.752120 28139 external_mini_cluster.cc:1428] Setting key 51cae1b0454c194ac5239340d2240161
Sep 28 16:34:38 dist-test-slave-g9sh krb5kdc[31679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613278, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:38.761711 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:38.749786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:36033 (local address 127.27.122.254:36541)
0928 16:34:38.749934 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:38.749939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:38.750747 (+ 808us) server_negotiation.cc:407] Connection header received
0928 16:34:38.752224 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:38.752227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:38.752287 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:38.752394 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:38.753836 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.754372 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.755503 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.755728 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.755837 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:38.758164 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:38.758174 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:38.758177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:38.758200 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:38.759553 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:38.760165 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:38.760169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:38.760171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:38.760258 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:38.760600 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:38.760605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:38.760607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:38.760785 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:38.760914 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:38.761428 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 16:34:38.761546 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:34:38.762460 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:38.750187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:36541 (local address 127.27.122.195:36033)
0928 16:34:38.750608 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:38.750650 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:38.752020 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:38.752624 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:38.752640 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:38.753141 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:38.753700 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:38.753715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.754513 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.754521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:38.755379 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:38.755388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.755874 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.755879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:38.756047 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:38.756556 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:38.756572 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:38.758008 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:38.759727 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:38.759733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:38.759751 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:38.760059 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:38.760353 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:38.760357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:38.760359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:38.760501 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:38.760944 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:38.760952 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:38.761185 (+ 233us) client_negotiation.cc:769] Sending connection context
0928 16:34:38.761374 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 16:34:38.761678 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":99,"threads_started":1}
I20260928 16:34:38.763499 32187 heartbeater.cc:347] Connected to a master server at 127.27.122.254:36541
I20260928 16:34:38.763703 32187 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:38.764119 32187 heartbeater.cc:511] Master 127.27.122.254:36541 requested a full tablet report, sending...
I20260928 16:34:38.765105 31724 ts_manager.cc:194] Registered new tserver with Master: 6b4fd68966fb4efd88ead93384cd43c4 (127.27.122.195:44575)
I20260928 16:34:38.765816 31724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:36033
I20260928 16:34:38.765959 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 16:34:38 dist-test-slave-g9sh krb5kdc[31679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:34:38 dist-test-slave-g9sh krb5kdc[31679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.193@KRBTEST.COM
I20260928 16:34:38.791589 32194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:38.782213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50554 (local address 127.27.122.193:37569)
0928 16:34:38.782489 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:38.782510 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:38.782538 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:34:38.782591 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:38.782596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:38.782765 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:38.782886 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:38.783545 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.784333 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.785298 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.785773 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.785872 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:38.788228 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:38.788239 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:38.788252 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:38.788279 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:38.789915 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:38.790328 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:38.790334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:38.790335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:38.790443 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:38.790696 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:38.790699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:38.790700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:38.790946 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:38.791119 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:38.791273 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 16:34:38.791432 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":100,"threads_started":1}
Sep 28 16:34:38 dist-test-slave-g9sh krb5kdc[31679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:38.802716 31763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:38.794530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46030 (local address 127.27.122.254:36541)
0928 16:34:38.794690 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:38.794693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:38.794708 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:34:38.794805 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:38.794807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:38.794846 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:38.794913 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:38.795494 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.796022 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.796778 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:38.797195 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:38.797297 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:38.799516 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:38.799520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:38.799523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:38.799555 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:38.800869 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:38.801317 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:38.801321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:38.801323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:38.801406 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:38.801685 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:38.801691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:38.801693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:38.802017 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:38.802169 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:38.802432 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 16:34:38.802547 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 16:34:38.803134 31724 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46030
I20260928 16:34:38.803730 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 31786
I20260928 16:34:38.811002 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 31922
I20260928 16:34:38.816359 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 32057
I20260928 16:34:38.821650 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 31692
2026-09-28T16:34:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2138 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:38 dist-test-slave-g9sh krb5kdc[32199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:38 dist-test-slave-g9sh krb5kdc[32199](info): set up 2 sockets
Sep 28 16:34:38 dist-test-slave-g9sh krb5kdc[32199](info): commencing operation
krb5kdc: starting...
W20260928 16:34:39.863505 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:39 dist-test-slave-g9sh krb5kdc[32199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:34:39.995152 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:36959
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:36959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:40.102604 32212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:40.102841 32212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:40.102975 32212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:40.106959 32212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:40.107009 32212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:40.107028 32212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:40.107088 32212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:40.107120 32212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:40.112126 32212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:36959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:36959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:40.113113 32212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:40.114110 32212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:40.122179 32220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:40.122390 32212 server_base.cc:1034] running on GCE node
W20260928 16:34:40.122268 32218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:40.122179 32217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:40.123016 32212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:40.124094 32212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:40.125269 32212 hybrid_clock.cc:648] HybridClock initialized: now 1790613280125239 us; error 43 us; skew 500 ppm
Sep 28 16:34:40 dist-test-slave-g9sh krb5kdc[32199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613280, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:40.127811 32212 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:34:40.128527 32212 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:40.128615 32212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:40.128835 32212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:40.130846 32212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "dc1f25a770da435ab70f4ba843d9514d"
format_stamp: "Formatted at 2026-09-28 16:34:40 on dist-test-slave-g9sh"
server_key: "c89e13a16941159dfcf2fcdf6bc7be2c"
server_key_iv: "15fc205a2b7d1fe242439b8868263c89"
server_key_version: "encryptionkey@0"
I20260928 16:34:40.131402 32212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "dc1f25a770da435ab70f4ba843d9514d"
format_stamp: "Formatted at 2026-09-28 16:34:40 on dist-test-slave-g9sh"
server_key: "c89e13a16941159dfcf2fcdf6bc7be2c"
server_key_iv: "15fc205a2b7d1fe242439b8868263c89"
server_key_version: "encryptionkey@0"
I20260928 16:34:40.134887 32212 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:34:40.137285 32226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:40.138399 32212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:40.138533 32212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "dc1f25a770da435ab70f4ba843d9514d"
format_stamp: "Formatted at 2026-09-28 16:34:40 on dist-test-slave-g9sh"
server_key: "c89e13a16941159dfcf2fcdf6bc7be2c"
server_key_iv: "15fc205a2b7d1fe242439b8868263c89"
server_key_version: "encryptionkey@0"
I20260928 16:34:40.138638 32212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:40.150279 32212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:40.152892 32212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:40.153113 32212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:40.159691 32212 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:36959
I20260928 16:34:40.159962 32279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:36959 every 8 connection(s)
I20260928 16:34:40.160369 32212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:40.162992 32280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:40.168267 32280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d: Bootstrap starting.
I20260928 16:34:40.170143 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 32212
I20260928 16:34:40.170281 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:40.170545 28139 external_mini_cluster.cc:1428] Setting key e2b4398b436b3fb7d6d8d6f541ed9406
I20260928 16:34:40.170789 32280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:40.171716 32280 log.cc:824] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:40.173688 32280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d: No bootstrap required, opened a new log
I20260928 16:34:40.177801 32280 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1f25a770da435ab70f4ba843d9514d" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36959 } }
I20260928 16:34:40.178079 32280 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:40.178150 32280 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1f25a770da435ab70f4ba843d9514d, State: Initialized, Role: FOLLOWER
I20260928 16:34:40.178670 32280 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1f25a770da435ab70f4ba843d9514d" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36959 } }
I20260928 16:34:40.178864 32280 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:40.178932 32280 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:40.179039 32280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [term 0 FOLLOWER]: Advancing to term 1
Sep 28 16:34:40 dist-test-slave-g9sh krb5kdc[32199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:40.179971 32280 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1f25a770da435ab70f4ba843d9514d" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36959 } }
I20260928 16:34:40.180305 32280 leader_election.cc:302] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc1f25a770da435ab70f4ba843d9514d; no voters:
I20260928 16:34:40.180609 32280 leader_election.cc:288] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:40.180723 32285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:40.181000 32285 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [term 1 LEADER]: Becoming Leader. State: Replica: dc1f25a770da435ab70f4ba843d9514d, State: Running, Role: LEADER
I20260928 16:34:40.181349 32285 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1f25a770da435ab70f4ba843d9514d" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36959 } }
I20260928 16:34:40.181994 32280 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:40.182473 32286 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc1f25a770da435ab70f4ba843d9514d. Latest consensus state: current_term: 1 leader_uuid: "dc1f25a770da435ab70f4ba843d9514d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1f25a770da435ab70f4ba843d9514d" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36959 } } }
I20260928 16:34:40.182587 32286 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:40.182976 32290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:40.182917 32287 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc1f25a770da435ab70f4ba843d9514d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc1f25a770da435ab70f4ba843d9514d" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36959 } } }
I20260928 16:34:40.184091 32287 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:40.186017 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.171678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51320 (local address 127.27.122.254:36959)
0928 16:34:40.172121 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:40.172136 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:40.172171 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:34:40.172790 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:40.172806 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:40.173193 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:40.173462 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:40.174358 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.175316 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.176520 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.177051 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.177181 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.180006 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:40.180012 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:40.180024 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:40.180075 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:40.182228 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:40.182899 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:40.182906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:40.182915 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:40.183064 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:40.183454 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:40.183457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:40.183459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:40.183973 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:40.184166 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:40.184725 (+ 559us) server_negotiation.cc:299] Negotiation successful
0928 16:34:40.185055 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":150,"threads_started":1}
I20260928 16:34:40.186368 32290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:40.192108 32290 catalog_manager.cc:1347] Generated new cluster ID: dba3f23359304b79b0cfa3eff3bd03b2
I20260928 16:34:40.192183 32290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:40.197365 32290 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:40.198199 32290 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:40.203210 32290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d: Generated new TSK 0
I20260928 16:34:40.203819 32290 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:34:40.232978 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36959
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:40.356104 32306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:40.356338 32306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:40.356381 32306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:40.360340 32306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:40.360390 32306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:40.360499 32306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:40.365525 32306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:40.366487 32306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:40.367478 32306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:40.376618 32314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:40.376641 32311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:40.376670 32312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:40.376822 32306 server_base.cc:1034] running on GCE node
I20260928 16:34:40.377390 32306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:40.378015 32306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:40.379209 32306 hybrid_clock.cc:648] HybridClock initialized: now 1790613280379192 us; error 30 us; skew 500 ppm
Sep 28 16:34:40 dist-test-slave-g9sh krb5kdc[32199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613280, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:40.382033 32306 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:34:40.382864 32306 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:40.382956 32306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:40.383224 32306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:40.385373 32306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1"
format_stamp: "Formatted at 2026-09-28 16:34:40 on dist-test-slave-g9sh"
server_key: "a952036d027719dfdbfb976592455645"
server_key_iv: "6b6ded321fa8d8e6a419c2c7b79010e5"
server_key_version: "encryptionkey@0"
I20260928 16:34:40.385939 32306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1"
format_stamp: "Formatted at 2026-09-28 16:34:40 on dist-test-slave-g9sh"
server_key: "a952036d027719dfdbfb976592455645"
server_key_iv: "6b6ded321fa8d8e6a419c2c7b79010e5"
server_key_version: "encryptionkey@0"
I20260928 16:34:40.389732 32306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:34:40.391999 32320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:40.393234 32306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:40.393373 32306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1"
format_stamp: "Formatted at 2026-09-28 16:34:40 on dist-test-slave-g9sh"
server_key: "a952036d027719dfdbfb976592455645"
server_key_iv: "6b6ded321fa8d8e6a419c2c7b79010e5"
server_key_version: "encryptionkey@0"
I20260928 16:34:40.393478 32306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:40.440147 32306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:40.443277 32306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:40.443481 32306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:40.444111 32306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:40.444988 32306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:40.445057 32306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:40.445128 32306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:40.445179 32306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:40.453969 32306 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:45887
I20260928 16:34:40.453996 32434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:45887 every 8 connection(s)
I20260928 16:34:40.454535 32306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:40.459842 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 32306
I20260928 16:34:40.459983 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:34:40.460366 28139 external_mini_cluster.cc:1428] Setting key 83782947285d33f5f1d1bd4fb86f7c6f
Sep 28 16:34:40 dist-test-slave-g9sh krb5kdc[32199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613280, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:40.468813 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.456190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:48765 (local address 127.27.122.254:36959)
0928 16:34:40.456329 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:40.456333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:40.457129 (+ 796us) server_negotiation.cc:407] Connection header received
0928 16:34:40.458409 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:40.458413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:40.458474 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:40.458576 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:40.459796 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.460539 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.461997 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.462236 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.462315 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.465052 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:40.465056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:40.465058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:40.465084 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:40.466650 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:40.467292 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:40.467295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:40.467296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:40.467374 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:40.467759 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:40.467762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:40.467764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:40.467946 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:40.468054 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:40.468523 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 16:34:40.468644 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:34:40.469635 32437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.456493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:36959 (local address 127.27.122.193:48765)
0928 16:34:40.456987 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:40.457030 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:40.458190 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:40.458758 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:40.458765 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:40.459109 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:40.459572 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:40.459620 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.460697 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.460701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:40.461862 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:40.461873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.462392 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.462399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:40.462649 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.463244 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:40.463270 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:40.464846 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:40.466771 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:40.466780 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:40.466797 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:40.467168 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:40.467479 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:40.467484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:40.467487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:40.467651 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:40.468053 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:40.468060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:40.468288 (+ 228us) client_negotiation.cc:769] Sending connection context
0928 16:34:40.468467 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 16:34:40.468777 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:34:40.470853 32435 heartbeater.cc:347] Connected to a master server at 127.27.122.254:36959
I20260928 16:34:40.471047 32435 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:40.471510 32435 heartbeater.cc:511] Master 127.27.122.254:36959 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
I20260928 16:34:40.473039 32243 ts_manager.cc:194] Registered new tserver with Master: 19db86d6f72f49a2b6afa2bfd77ac4e1 (127.27.122.193:45887)
I20260928 16:34:40.474421 32243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:48765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:34:40.489637 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36959
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:40.600411 32440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:40.600636 32440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:40.600678 32440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:40.604568 32440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:40.604615 32440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:40.604689 32440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:40.609694 32440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:40.610574 32440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:40.611471 32440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:40.620846 32445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:40.620823 32446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:40.620980 32440 server_base.cc:1034] running on GCE node
W20260928 16:34:40.620817 32448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:40.621524 32440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:40.622104 32440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:40.623263 32440 hybrid_clock.cc:648] HybridClock initialized: now 1790613280623250 us; error 25 us; skew 500 ppm
Sep 28 16:34:40 dist-test-slave-g9sh krb5kdc[32199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613280, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:40.625900 32440 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:34:40.626580 32440 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:40.626675 32440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:40.626909 32440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:40.628811 32440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "6e03de0f90ab418cb8702fd94350722d"
format_stamp: "Formatted at 2026-09-28 16:34:40 on dist-test-slave-g9sh"
server_key: "c8bccba4621ff3b11b4cd2b5ce60013d"
server_key_iv: "440d73a68b8e0b0632c8a36922977e03"
server_key_version: "encryptionkey@0"
I20260928 16:34:40.629413 32440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "6e03de0f90ab418cb8702fd94350722d"
format_stamp: "Formatted at 2026-09-28 16:34:40 on dist-test-slave-g9sh"
server_key: "c8bccba4621ff3b11b4cd2b5ce60013d"
server_key_iv: "440d73a68b8e0b0632c8a36922977e03"
server_key_version: "encryptionkey@0"
I20260928 16:34:40.632870 32440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:34:40.635130 32454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:40.636277 32440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:40.636407 32440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "6e03de0f90ab418cb8702fd94350722d"
format_stamp: "Formatted at 2026-09-28 16:34:40 on dist-test-slave-g9sh"
server_key: "c8bccba4621ff3b11b4cd2b5ce60013d"
server_key_iv: "440d73a68b8e0b0632c8a36922977e03"
server_key_version: "encryptionkey@0"
I20260928 16:34:40.636513 32440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:40.650034 32440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:40.652561 32440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:40.652746 32440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:40.653395 32440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:40.654165 32440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:40.654234 32440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:40.654299 32440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:40.654350 32440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:40.662569 32440 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:45221
I20260928 16:34:40.662642 32567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:45221 every 8 connection(s)
I20260928 16:34:40.663144 32440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:40.665387 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 32440
I20260928 16:34:40.665529 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:34:40.665828 28139 external_mini_cluster.cc:1428] Setting key e296e18e4835d99b3166f89fe44a2b17
Sep 28 16:34:40 dist-test-slave-g9sh krb5kdc[32199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613280, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
I20260928 16:34:40.677150 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.664774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:42781 (local address 127.27.122.254:36959)
0928 16:34:40.664962 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:40.664968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:40.666192 (+ 1224us) server_negotiation.cc:407] Connection header received
0928 16:34:40.667041 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:40.667045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:40.667113 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:40.667210 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:40.668321 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.669171 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.670287 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.670694 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.670821 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.673360 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:40.673366 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:40.673371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:40.673402 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:40.675032 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:40.675634 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:40.675637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:40.675638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:40.675712 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:40.676052 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:40.676054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:40.676055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:40.676224 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:40.676350 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:40.676835 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 16:34:40.676973 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
I20260928 16:34:40.678013 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.665158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:36959 (local address 127.27.122.194:42781)
0928 16:34:40.666032 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:40.666083 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:40.666842 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:40.667385 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:40.667392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:40.667745 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:40.668179 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:40.668193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.669300 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.669303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:40.670162 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:40.670171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.670910 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.670917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:40.671107 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.671692 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:40.671707 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:40.673156 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:40.675223 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:40.675228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:40.675239 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:40.675526 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:40.675818 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:40.675820 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:40.675822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:40.675964 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:40.676362 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:40.676370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:40.676601 (+ 231us) client_negotiation.cc:769] Sending connection context
0928 16:34:40.676790 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 16:34:40.677120 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":134,"threads_started":1}
I20260928 16:34:40.679188 32569 heartbeater.cc:347] Connected to a master server at 127.27.122.254:36959
I20260928 16:34:40.679457 32569 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:40.679901 32569 heartbeater.cc:511] Master 127.27.122.254:36959 requested a full tablet report, sending...
I20260928 16:34:40.681003 32243 ts_manager.cc:194] Registered new tserver with Master: 6e03de0f90ab418cb8702fd94350722d (127.27.122.194:45221)
I20260928 16:34:40.681867 32243 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:42781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:34:40.693459 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36959
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:40.800235 32574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:40.800479 32574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:40.800522 32574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:40.804380 32574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:40.804425 32574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:40.804481 32574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:40.809458 32574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:40.810364 32574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:40.811288 32574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:40.820814 32579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:40.820822 32582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:40.820884 32574 server_base.cc:1034] running on GCE node
W20260928 16:34:40.820803 32580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:40.821524 32574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:40.822054 32574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:40.823211 32574 hybrid_clock.cc:648] HybridClock initialized: now 1790613280823192 us; error 29 us; skew 500 ppm
Sep 28 16:34:40 dist-test-slave-g9sh krb5kdc[32199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613280, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:40.825795 32574 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:34:40.826462 32574 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:40.826556 32574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:40.826786 32574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:40.828702 32574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "df7b60fe76034ecda6e6b78a5bbcc23b"
format_stamp: "Formatted at 2026-09-28 16:34:40 on dist-test-slave-g9sh"
server_key: "0a441290d4cea0b329648735cb80bdac"
server_key_iv: "48ee8b5b357222240d4edd51f3d93ed5"
server_key_version: "encryptionkey@0"
I20260928 16:34:40.829303 32574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "df7b60fe76034ecda6e6b78a5bbcc23b"
format_stamp: "Formatted at 2026-09-28 16:34:40 on dist-test-slave-g9sh"
server_key: "0a441290d4cea0b329648735cb80bdac"
server_key_iv: "48ee8b5b357222240d4edd51f3d93ed5"
server_key_version: "encryptionkey@0"
I20260928 16:34:40.832729 32574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:34:40.835114 32588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:40.836262 32574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:40.836393 32574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "df7b60fe76034ecda6e6b78a5bbcc23b"
format_stamp: "Formatted at 2026-09-28 16:34:40 on dist-test-slave-g9sh"
server_key: "0a441290d4cea0b329648735cb80bdac"
server_key_iv: "48ee8b5b357222240d4edd51f3d93ed5"
server_key_version: "encryptionkey@0"
I20260928 16:34:40.836501 32574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:40.854228 32574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:40.856804 32574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:40.857012 32574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:40.857551 32574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:40.858328 32574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:40.858405 32574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:40.858475 32574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:40.858512 32574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:40.867295 32574 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:45787
I20260928 16:34:40.867318 32702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:45787 every 8 connection(s)
I20260928 16:34:40.867841 32574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:34:40.868963 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 32574
I20260928 16:34:40.869112 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:34:40.869441 28139 external_mini_cluster.cc:1428] Setting key 206e38bafee48a99034ead1fe1aa9786
Sep 28 16:34:40 dist-test-slave-g9sh krb5kdc[32199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613280, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:40.880837 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.869612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:45255 (local address 127.27.122.254:36959)
0928 16:34:40.869778 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:40.869783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:40.870670 (+ 887us) server_negotiation.cc:407] Connection header received
0928 16:34:40.872009 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:40.872011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:40.872057 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:40.872125 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:40.873227 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.873753 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.874823 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.875032 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.875104 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.877634 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:40.877637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:40.877639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:40.877663 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:40.879066 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:40.879560 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:40.879563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:40.879564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:40.879643 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:40.879947 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:40.879949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:40.879950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:40.880116 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:40.880196 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:40.880631 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 16:34:40.880720 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:34:40.881701 32705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.869968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:36959 (local address 127.27.122.195:45255)
0928 16:34:40.870512 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:40.870569 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:40.871821 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:40.872257 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:40.872263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:40.872581 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:40.873107 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:40.873123 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.873861 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.873863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:40.874722 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:40.874730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.875150 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.875153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:40.875282 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.875945 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:40.875961 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:40.877482 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:40.879202 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:40.879206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:40.879217 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:40.879471 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:40.879724 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:40.879726 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:40.879728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:40.879851 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:40.880207 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:40.880211 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:40.880423 (+ 212us) client_negotiation.cc:769] Sending connection context
0928 16:34:40.880573 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 16:34:40.880798 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":158,"threads_started":1}
I20260928 16:34:40.882942 32703 heartbeater.cc:347] Connected to a master server at 127.27.122.254:36959
I20260928 16:34:40.883170 32703 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:40.883702 32703 heartbeater.cc:511] Master 127.27.122.254:36959 requested a full tablet report, sending...
I20260928 16:34:40.884698 32242 ts_manager.cc:194] Registered new tserver with Master: df7b60fe76034ecda6e6b78a5bbcc23b (127.27.122.195:45787)
I20260928 16:34:40.885344 32242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:45255
I20260928 16:34:40.895017 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:34:40.903654 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.896943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.27.122.254:36959)
0928 16:34:40.897051 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:40.897055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:40.897230 (+ 175us) server_negotiation.cc:407] Connection header received
0928 16:34:40.897320 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:40.897323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:40.897359 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:40.897427 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:40.898149 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.898640 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.899433 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.899641 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.899713 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.901082 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:40.901085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:40.901087 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:40.901107 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:40.902297 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:40.902670 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:40.902672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:40.902674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:40.902743 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:40.902968 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:40.902970 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:40.902971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:40.903165 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:40.903263 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:40.903452 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 16:34:40.903536 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 16:34:40.906361 32242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:34:40.909330 32242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:40.923473 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.917690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:45221 (local address 127.0.0.1:36014)
0928 16:34:40.918267 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:40.918280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:40.918370 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:40.919357 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:40.919360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:40.919373 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:40.919475 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:40.919483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.920511 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.920517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:40.922041 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:40.922052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.922939 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.922943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:40.923049 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.923084 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:34:40.923196 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 16:34:40.923308 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":102,"thread_start_us":91,"threads_started":1}
I20260928 16:34:40.923606 32714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.917461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:45787 (local address 127.0.0.1:57602)
0928 16:34:40.917809 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:40.917860 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:40.917977 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:40.919254 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:40.919261 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:40.919284 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:40.919402 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:40.919409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.920834 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.920838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:40.922041 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:40.922052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.923157 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.923163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:40.923263 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.923285 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:34:40.923333 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 16:34:40.923427 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":32,"thread_start_us":78,"threads_started":1}
I20260928 16:34:40.923669 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.918571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34370 (local address 127.27.122.193:45887)
0928 16:34:40.918948 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:40.918954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:40.918972 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:34:40.919038 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:40.919042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:40.919238 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:40.919705 (+ 467us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:40.919864 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.920948 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.922336 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.923112 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.923223 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.923347 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:40.923430 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 16:34:40.923530 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":111,"threads_started":1}
I20260928 16:34:40.924085 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.918230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.27.122.194:45221)
0928 16:34:40.918686 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:40.918699 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:40.918723 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:34:40.918799 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:40.918805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:40.919021 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:40.919188 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:40.919602 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.920376 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.922167 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.922809 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.922951 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.923098 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:40.923808 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 16:34:40.923932 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":105,"threads_started":1}
I20260928 16:34:40.924096 32716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.917863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:45887 (local address 127.0.0.1:34370)
0928 16:34:40.918299 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:40.918313 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:40.918376 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:40.919388 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:40.919394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:40.919426 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:40.919551 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:40.919559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.921078 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.921084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:40.922213 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:40.922225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.923223 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.923226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:40.923305 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.923325 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 16:34:40.923401 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 16:34:40.923456 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":39,"spinlock_wait_cycles":896,"thread_start_us":83,"threads_started":1}
I20260928 16:34:40.924378 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:40.918138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.27.122.195:45787)
0928 16:34:40.918745 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:40.918757 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:40.918779 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:34:40.918854 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:40.918859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:40.919049 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:40.919257 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:40.919529 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.920692 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.922200 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:40.923012 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:40.923563 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:40.923696 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:40.924000 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 16:34:40.924078 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":128,"threads_started":1}
I20260928 16:34:40.926790 32636 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac747c5bc354ba59e73743ffba6484e (DEFAULT_TABLE table=test-table [id=343893e5165c48e78939b41c563d463c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:40.926908 32502 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac747c5bc354ba59e73743ffba6484e (DEFAULT_TABLE table=test-table [id=343893e5165c48e78939b41c563d463c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:40.926910 32368 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac747c5bc354ba59e73743ffba6484e (DEFAULT_TABLE table=test-table [id=343893e5165c48e78939b41c563d463c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:40.927802 32636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac747c5bc354ba59e73743ffba6484e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:40.927888 32502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac747c5bc354ba59e73743ffba6484e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:40.927897 32368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac747c5bc354ba59e73743ffba6484e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:40.934353 32720 tablet_bootstrap.cc:492] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b: Bootstrap starting.
I20260928 16:34:40.935364 32722 tablet_bootstrap.cc:492] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1: Bootstrap starting.
I20260928 16:34:40.936444 32720 tablet_bootstrap.cc:654] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:40.937071 32722 tablet_bootstrap.cc:654] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:40.937589 32720 log.cc:824] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:40.938141 32722 log.cc:824] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:40.938190 32721 tablet_bootstrap.cc:492] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d: Bootstrap starting.
I20260928 16:34:40.939527 32720 tablet_bootstrap.cc:492] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b: No bootstrap required, opened a new log
I20260928 16:34:40.939715 32720 ts_tablet_manager.cc:1397] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:34:40.940148 32722 tablet_bootstrap.cc:492] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1: No bootstrap required, opened a new log
I20260928 16:34:40.940399 32722 ts_tablet_manager.cc:1397] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260928 16:34:40.940589 32721 tablet_bootstrap.cc:654] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:40.941665 32721 log.cc:824] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:40.943439 32721 tablet_bootstrap.cc:492] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d: No bootstrap required, opened a new log
I20260928 16:34:40.943620 32721 ts_tablet_manager.cc:1397] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:34:40.943717 32720 raft_consensus.cc:348] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7b60fe76034ecda6e6b78a5bbcc23b" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 45787 } } peers { permanent_uuid: "6e03de0f90ab418cb8702fd94350722d" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45221 } } peers { permanent_uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45887 } }
I20260928 16:34:40.944001 32720 raft_consensus.cc:374] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:40.944058 32720 raft_consensus.cc:729] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df7b60fe76034ecda6e6b78a5bbcc23b, State: Initialized, Role: FOLLOWER
I20260928 16:34:40.944108 32722 raft_consensus.cc:348] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7b60fe76034ecda6e6b78a5bbcc23b" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 45787 } } peers { permanent_uuid: "6e03de0f90ab418cb8702fd94350722d" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45221 } } peers { permanent_uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45887 } }
I20260928 16:34:40.944346 32722 raft_consensus.cc:374] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:40.944471 32722 raft_consensus.cc:729] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19db86d6f72f49a2b6afa2bfd77ac4e1, State: Initialized, Role: FOLLOWER
I20260928 16:34:40.944496 32720 consensus_queue.cc:260] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7b60fe76034ecda6e6b78a5bbcc23b" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 45787 } } peers { permanent_uuid: "6e03de0f90ab418cb8702fd94350722d" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45221 } } peers { permanent_uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45887 } }
I20260928 16:34:40.944975 32722 consensus_queue.cc:260] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7b60fe76034ecda6e6b78a5bbcc23b" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 45787 } } peers { permanent_uuid: "6e03de0f90ab418cb8702fd94350722d" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45221 } } peers { permanent_uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45887 } }
I20260928 16:34:40.945142 32703 heartbeater.cc:503] Master 127.27.122.254:36959 was elected leader, sending a full tablet report...
I20260928 16:34:40.945497 32720 ts_tablet_manager.cc:1428] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 16:34:40.945922 32722 ts_tablet_manager.cc:1428] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:34:40.946223 32435 heartbeater.cc:503] Master 127.27.122.254:36959 was elected leader, sending a full tablet report...
I20260928 16:34:40.947561 32721 raft_consensus.cc:348] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7b60fe76034ecda6e6b78a5bbcc23b" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 45787 } } peers { permanent_uuid: "6e03de0f90ab418cb8702fd94350722d" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45221 } } peers { permanent_uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45887 } }
I20260928 16:34:40.947851 32721 raft_consensus.cc:374] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:40.947908 32721 raft_consensus.cc:729] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e03de0f90ab418cb8702fd94350722d, State: Initialized, Role: FOLLOWER
I20260928 16:34:40.948371 32721 consensus_queue.cc:260] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7b60fe76034ecda6e6b78a5bbcc23b" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 45787 } } peers { permanent_uuid: "6e03de0f90ab418cb8702fd94350722d" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45221 } } peers { permanent_uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45887 } }
I20260928 16:34:40.949174 32569 heartbeater.cc:503] Master 127.27.122.254:36959 was elected leader, sending a full tablet report...
I20260928 16:34:40.949251 32721 ts_tablet_manager.cc:1428] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260928 16:34:40.955513 32436 tablet.cc:2369] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:41.087836 32728 raft_consensus.cc:482] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:41.088068 32728 raft_consensus.cc:504] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7b60fe76034ecda6e6b78a5bbcc23b" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 45787 } } peers { permanent_uuid: "6e03de0f90ab418cb8702fd94350722d" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45221 } } peers { permanent_uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45887 } }
I20260928 16:34:41.089380 32728 leader_election.cc:288] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df7b60fe76034ecda6e6b78a5bbcc23b (127.27.122.195:45787), 19db86d6f72f49a2b6afa2bfd77ac4e1 (127.27.122.193:45887)
I20260928 16:34:41.094398 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.089732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:54111 (local address 127.27.122.195:45787)
0928 16:34:41.089861 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:41.089865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:41.089880 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:34:41.090041 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:41.090044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:41.090090 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:41.090192 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:41.090470 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:41.091264 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:41.093031 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:41.093640 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:41.093772 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:41.093826 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:41.094196 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 16:34:41.094267 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:34:41.094398 32719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.089788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:59965 (local address 127.27.122.193:45887)
0928 16:34:41.089936 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:41.089940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:41.090178 (+ 238us) server_negotiation.cc:407] Connection header received
0928 16:34:41.090317 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:41.090320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:41.090366 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:41.090493 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:41.090731 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:41.091448 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:41.093031 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:41.093648 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:41.093772 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:41.093826 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:41.094196 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 16:34:41.094267 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:34:41.094442 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.089582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:45787 (local address 127.27.122.194:54111)
0928 16:34:41.089771 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:41.089789 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:41.089888 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:41.090207 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:41.090210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:41.090224 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:41.090357 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:41.090362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:41.091371 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:41.091373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:41.092854 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:41.092865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:41.093789 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:41.093793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:41.094021 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:41.094083 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 16:34:41.094167 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:34:41.094274 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":184}
I20260928 16:34:41.094451 32729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.089694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:45887 (local address 127.27.122.194:59965)
0928 16:34:41.090078 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:41.090088 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:41.090206 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:41.090516 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:41.090518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:41.090534 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:41.090624 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:41.090629 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:41.091546 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:41.091548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:41.092854 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:41.092865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:41.093788 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:41.093793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:41.094011 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:41.094083 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 16:34:41.094167 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:34:41.094268 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":103,"thread_start_us":129,"threads_started":1}
I20260928 16:34:41.095252 32657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac747c5bc354ba59e73743ffba6484e" candidate_uuid: "6e03de0f90ab418cb8702fd94350722d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7b60fe76034ecda6e6b78a5bbcc23b" is_pre_election: true
I20260928 16:34:41.095257 32388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac747c5bc354ba59e73743ffba6484e" candidate_uuid: "6e03de0f90ab418cb8702fd94350722d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1" is_pre_election: true
I20260928 16:34:41.095551 32388 raft_consensus.cc:2454] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e03de0f90ab418cb8702fd94350722d in term 0.
I20260928 16:34:41.095583 32657 raft_consensus.cc:2454] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e03de0f90ab418cb8702fd94350722d in term 0.
I20260928 16:34:41.096037 32457 leader_election.cc:302] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19db86d6f72f49a2b6afa2bfd77ac4e1, 6e03de0f90ab418cb8702fd94350722d; no voters:
I20260928 16:34:41.096304 32728 raft_consensus.cc:2781] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:41.096405 32728 raft_consensus.cc:482] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:41.096462 32728 raft_consensus.cc:3037] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:41.097572 32728 raft_consensus.cc:504] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7b60fe76034ecda6e6b78a5bbcc23b" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 45787 } } peers { permanent_uuid: "6e03de0f90ab418cb8702fd94350722d" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45221 } } peers { permanent_uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45887 } }
I20260928 16:34:41.097882 32728 leader_election.cc:288] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [CANDIDATE]: Term 1 election: Requested vote from peers df7b60fe76034ecda6e6b78a5bbcc23b (127.27.122.195:45787), 19db86d6f72f49a2b6afa2bfd77ac4e1 (127.27.122.193:45887)
I20260928 16:34:41.098196 32657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac747c5bc354ba59e73743ffba6484e" candidate_uuid: "6e03de0f90ab418cb8702fd94350722d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df7b60fe76034ecda6e6b78a5bbcc23b"
I20260928 16:34:41.098201 32388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac747c5bc354ba59e73743ffba6484e" candidate_uuid: "6e03de0f90ab418cb8702fd94350722d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1"
I20260928 16:34:41.098316 32388 raft_consensus.cc:3037] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:41.098316 32657 raft_consensus.cc:3037] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:41.099604 32657 raft_consensus.cc:2454] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e03de0f90ab418cb8702fd94350722d in term 1.
I20260928 16:34:41.099609 32388 raft_consensus.cc:2454] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e03de0f90ab418cb8702fd94350722d in term 1.
I20260928 16:34:41.099907 32458 leader_election.cc:302] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 19db86d6f72f49a2b6afa2bfd77ac4e1, 6e03de0f90ab418cb8702fd94350722d, df7b60fe76034ecda6e6b78a5bbcc23b; no voters:
I20260928 16:34:41.100113 32728 raft_consensus.cc:2781] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:41.100307 32728 raft_consensus.cc:686] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [term 1 LEADER]: Becoming Leader. State: Replica: 6e03de0f90ab418cb8702fd94350722d, State: Running, Role: LEADER
I20260928 16:34:41.100618 32728 consensus_queue.cc:237] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7b60fe76034ecda6e6b78a5bbcc23b" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 45787 } } peers { permanent_uuid: "6e03de0f90ab418cb8702fd94350722d" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45221 } } peers { permanent_uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45887 } }
I20260928 16:34:41.103924 32241 catalog_manager.cc:5495] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e03de0f90ab418cb8702fd94350722d (127.27.122.194). New cstate: current_term: 1 leader_uuid: "6e03de0f90ab418cb8702fd94350722d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7b60fe76034ecda6e6b78a5bbcc23b" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 45787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e03de0f90ab418cb8702fd94350722d" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45887 } health_report { overall_health: UNKNOWN } } }
W20260928 16:34:41.118630 32704 tablet.cc:2369] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:41.156140 32718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.152837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36016 (local address 127.27.122.194:45221)
0928 16:34:41.152990 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:41.152994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:41.153008 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:34:41.153142 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:41.153144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:41.153202 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:41.153301 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:41.153525 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:41.154026 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:41.154826 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:41.155032 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:41.155109 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:41.155371 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:41.155816 (+ 445us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:41.155909 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:41.155979 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 16:34:41.156027 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:34:41.162007 32657 raft_consensus.cc:1261] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b [term 1 FOLLOWER]: Refusing update from remote peer 6e03de0f90ab418cb8702fd94350722d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:41.162056 32388 raft_consensus.cc:1261] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1 [term 1 FOLLOWER]: Refusing update from remote peer 6e03de0f90ab418cb8702fd94350722d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:41.162751 32730 consensus_queue.cc:1035] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19db86d6f72f49a2b6afa2bfd77ac4e1" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:41.162947 32728 consensus_queue.cc:1035] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df7b60fe76034ecda6e6b78a5bbcc23b" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 45787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:34:41.164252 32570 tablet.cc:2369] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:41.171124 32733 mvcc.cc:205] Tried to move back new op lower bound from 7334351999632257024 to 7334351999390982144. Current Snapshot: MvccSnapshot[applied={T|T < 7334351999632257024}]
I20260928 16:34:41.171550 32735 mvcc.cc:205] Tried to move back new op lower bound from 7334351999632257024 to 7334351999390982144. Current Snapshot: MvccSnapshot[applied={T|T < 7334351999632257024}]
I20260928 16:34:41.173130 32734 mvcc.cc:205] Tried to move back new op lower bound from 7334351999632257024 to 7334351999390982144. Current Snapshot: MvccSnapshot[applied={T|T < 7334351999632257024}]
I20260928 16:34:41.182174 32242 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:41.182317 32242 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:41.184695 32242 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc1f25a770da435ab70f4ba843d9514d: Sending DeleteTablet for 3 replicas of tablet 8ac747c5bc354ba59e73743ffba6484e
I20260928 16:34:41.185461 32636 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac747c5bc354ba59e73743ffba6484e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:41 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:57602
I20260928 16:34:41.185490 32502 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac747c5bc354ba59e73743ffba6484e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:41 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:36014
I20260928 16:34:41.185968 32745 tablet_replica.cc:330] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b: stopping tablet replica
I20260928 16:34:41.186340 32745 raft_consensus.cc:2229] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:41.186369 32368 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac747c5bc354ba59e73743ffba6484e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:41 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:34370
I20260928 16:34:41.186668 32745 raft_consensus.cc:2258] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:41.186879 32746 tablet_replica.cc:330] T 8ac747c5bc354ba59e73743ffba6484e P 19db86d6f72f49a2b6afa2bfd77ac4e1: stopping tablet replica
I20260928 16:34:41.186973 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 32306
I20260928 16:34:41.188135 32745 ts_tablet_manager.cc:1905] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:41.191015 32745 ts_tablet_manager.cc:1918] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:41.191111 32745 log.cc:1196] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/8ac747c5bc354ba59e73743ffba6484e
I20260928 16:34:41.191435 32745 ts_tablet_manager.cc:1939] T 8ac747c5bc354ba59e73743ffba6484e P df7b60fe76034ecda6e6b78a5bbcc23b: Deleting consensus metadata
I20260928 16:34:41.192554 32744 tablet_replica.cc:330] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d: stopping tablet replica
I20260928 16:34:41.192760 32230 catalog_manager.cc:4841] TS df7b60fe76034ecda6e6b78a5bbcc23b (127.27.122.195:45787): tablet 8ac747c5bc354ba59e73743ffba6484e (table test-table [id=343893e5165c48e78939b41c563d463c]) successfully deleted
I20260928 16:34:41.192857 32744 raft_consensus.cc:2229] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:41.193238 32744 raft_consensus.cc:2258] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:41.194463 32744 ts_tablet_manager.cc:1905] T 8ac747c5bc354ba59e73743ffba6484e P 6e03de0f90ab418cb8702fd94350722d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 16:34:41.195076 32229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:45887 (error 108)
I20260928 16:34:41.196050 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.195736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:45887 (local address 127.0.0.1:34386)
0928 16:34:41.195877 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:41.195951 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:45887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 16:34:41.196125 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 32440
W20260928 16:34:41.197283 32229 catalog_manager.cc:4584] TS 19db86d6f72f49a2b6afa2bfd77ac4e1 (127.27.122.193:45887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ac747c5bc354ba59e73743ffba6484e: Network error: Client connection negotiation failed: client connection to 127.27.122.193:45887: connect: Connection refused (error 111)
I20260928 16:34:41.202100 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 32574
I20260928 16:34:41.203230 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.202234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:45221 (local address 127.0.0.1:36032)
0928 16:34:41.203065 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:41.203139 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.194:45221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":736}
I20260928 16:34:41.207718 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 32212
2026-09-28T16:34:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2385 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T16:34:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:41Z Disabled control of system clock
I20260928 16:34:41.238042 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:36227
--webserver_interface=127.27.122.254
--webserver_port=0
--builtin_ntp_servers=127.27.122.212:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:36227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 16:34:41.349911 32751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:41.350167 32751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:41.350205 32751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:41.354173 32751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:41.354223 32751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:41.354238 32751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:41.354254 32751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:41.354269 32751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:41.358819 32751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:36227
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:36227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:41.359800 32751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:41.361172 32751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:41.369544 32759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:41.369535 32757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:41.369535 32756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:41.369633 32751 server_base.cc:1034] running on GCE node
I20260928 16:34:41.370358 32751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:41.371373 32751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:41.372534 32751 hybrid_clock.cc:648] HybridClock initialized: now 1790613281372491 us; error 56 us; skew 500 ppm
I20260928 16:34:41.374286 32751 webserver.cc:460] Webserver started at http://127.27.122.254:40967/ using document root <none> and password file <none>
I20260928 16:34:41.374836 32751 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:41.374902 32751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:41.375077 32751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:41.377103 32751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "10ea0a98604b4b30884d50b6fc783e4a"
format_stamp: "Formatted at 2026-09-28 16:34:41 on dist-test-slave-g9sh"
server_key: "2221a1dd1efe9d02d6e596c209343fdd"
server_key_iv: "f1066d498e2f683747ceac35bef730b5"
server_key_version: "encryptionkey@0"
I20260928 16:34:41.377628 32751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "10ea0a98604b4b30884d50b6fc783e4a"
format_stamp: "Formatted at 2026-09-28 16:34:41 on dist-test-slave-g9sh"
server_key: "2221a1dd1efe9d02d6e596c209343fdd"
server_key_iv: "f1066d498e2f683747ceac35bef730b5"
server_key_version: "encryptionkey@0"
I20260928 16:34:41.381227 32751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:34:41.383546 32765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:41.384747 32751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:41.384945 32751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "10ea0a98604b4b30884d50b6fc783e4a"
format_stamp: "Formatted at 2026-09-28 16:34:41 on dist-test-slave-g9sh"
server_key: "2221a1dd1efe9d02d6e596c209343fdd"
server_key_iv: "f1066d498e2f683747ceac35bef730b5"
server_key_version: "encryptionkey@0"
I20260928 16:34:41.385071 32751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:41.389683 32751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:41.390268 32751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:41.390450 32751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:41.396795 32751 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:36227
I20260928 16:34:41.396803 349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:36227 every 8 connection(s)
I20260928 16:34:41.397708 32751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:41.400751 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:41.403115 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 32751
I20260928 16:34:41.403237 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:41.403544 28139 external_mini_cluster.cc:1428] Setting key 080b8bf734d4b728fccfbce8231e15f7
I20260928 16:34:41.406857 350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a: Bootstrap starting.
I20260928 16:34:41.409312 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.404990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.27.122.254:36227)
0928 16:34:41.405617 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:41.405631 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:41.405664 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:34:41.406267 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:41.406283 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:41.406630 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:41.406964 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:41.407745 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:41.407750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:41.407761 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:41.407772 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:41.407827 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:41.407843 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:41.407990 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:41.408063 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 16:34:41.408310 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":140,"threads_started":1}
I20260928 16:34:41.409493 350 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:41.410368 350 log.cc:824] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:41.413195 350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a: No bootstrap required, opened a new log
I20260928 16:34:41.417670 350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ea0a98604b4b30884d50b6fc783e4a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36227 } }
I20260928 16:34:41.417867 350 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:41.417933 350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10ea0a98604b4b30884d50b6fc783e4a, State: Initialized, Role: FOLLOWER
I20260928 16:34:41.418402 350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ea0a98604b4b30884d50b6fc783e4a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36227 } }
I20260928 16:34:41.418548 350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:41.418620 350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:41.418713 350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:41.419662 350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ea0a98604b4b30884d50b6fc783e4a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36227 } }
I20260928 16:34:41.420025 350 leader_election.cc:302] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10ea0a98604b4b30884d50b6fc783e4a; no voters:
I20260928 16:34:41.420331 350 leader_election.cc:288] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:41.420444 355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:41.420718 355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [term 1 LEADER]: Becoming Leader. State: Replica: 10ea0a98604b4b30884d50b6fc783e4a, State: Running, Role: LEADER
I20260928 16:34:41.421028 355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ea0a98604b4b30884d50b6fc783e4a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36227 } }
I20260928 16:34:41.421545 350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:41.422569 357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10ea0a98604b4b30884d50b6fc783e4a. Latest consensus state: current_term: 1 leader_uuid: "10ea0a98604b4b30884d50b6fc783e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ea0a98604b4b30884d50b6fc783e4a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36227 } } }
I20260928 16:34:41.422701 357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:41.423069 364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:41.423108 356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10ea0a98604b4b30884d50b6fc783e4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10ea0a98604b4b30884d50b6fc783e4a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36227 } } }
I20260928 16:34:41.423322 356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:41.426172 364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:41.431592 364 catalog_manager.cc:1347] Generated new cluster ID: f6d13e3b1efc4dcabbb062544098296d
I20260928 16:34:41.431658 364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:41.441881 364 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:41.442169 364 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:41.451586 364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a: Generated new TSK 0
I20260928 16:34:41.452194 364 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:34:41.458447 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36227
--builtin_ntp_servers=127.27.122.212:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 16:34:41.569820 374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:41.570046 374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:41.570084 374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:41.574108 374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:41.574156 374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:41.574210 374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:41.578896 374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:41.579953 374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:41.581382 374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:41.590880 379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:41.591070 374 server_base.cc:1034] running on GCE node
W20260928 16:34:41.590873 382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:41.590873 380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:41.591558 374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:41.592180 374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:41.593366 374 hybrid_clock.cc:648] HybridClock initialized: now 1790613281593335 us; error 42 us; skew 500 ppm
I20260928 16:34:41.595180 374 webserver.cc:460] Webserver started at http://127.27.122.193:38839/ using document root <none> and password file <none>
I20260928 16:34:41.595758 374 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:41.595852 374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:41.596092 374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:41.598678 374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "f7fce234f8cc449eb9e411b156ba3303"
format_stamp: "Formatted at 2026-09-28 16:34:41 on dist-test-slave-g9sh"
server_key: "c6c5e64caaaa0955c913e8874b9cfdd2"
server_key_iv: "bd6ce1ff2c1a98c5ac71c757fdc745c0"
server_key_version: "encryptionkey@0"
I20260928 16:34:41.599329 374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "f7fce234f8cc449eb9e411b156ba3303"
format_stamp: "Formatted at 2026-09-28 16:34:41 on dist-test-slave-g9sh"
server_key: "c6c5e64caaaa0955c913e8874b9cfdd2"
server_key_iv: "bd6ce1ff2c1a98c5ac71c757fdc745c0"
server_key_version: "encryptionkey@0"
I20260928 16:34:41.603207 374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:34:41.605626 388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:41.606765 374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:41.606901 374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "f7fce234f8cc449eb9e411b156ba3303"
format_stamp: "Formatted at 2026-09-28 16:34:41 on dist-test-slave-g9sh"
server_key: "c6c5e64caaaa0955c913e8874b9cfdd2"
server_key_iv: "bd6ce1ff2c1a98c5ac71c757fdc745c0"
server_key_version: "encryptionkey@0"
I20260928 16:34:41.607009 374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:41.611428 374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:41.612010 374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:41.612200 374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:41.612744 374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:41.613684 374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:41.613757 374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:41.613823 374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:41.613875 374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:41.622558 374 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:41087
I20260928 16:34:41.622589 502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:41087 every 8 connection(s)
I20260928 16:34:41.623462 374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:41.624464 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 374
I20260928 16:34:41.624603 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:34:41.624934 28139 external_mini_cluster.cc:1428] Setting key ecefcc668080237fe339c2ad61b6d7f8
I20260928 16:34:41.626962 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36227
--builtin_ntp_servers=127.27.122.212:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 16:34:41.629805 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.625307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:38601 (local address 127.27.122.254:36227)
0928 16:34:41.625432 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:41.625442 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:41.626660 (+ 1218us) server_negotiation.cc:407] Connection header received
0928 16:34:41.628056 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:41.628059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:41.628102 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:41.628179 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:41.629307 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:41.629310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:41.629313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:41.629325 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:41.629347 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:41.629356 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:41.629457 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:41.629631 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 16:34:41.629677 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 16:34:41.630739 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.625752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:36227 (local address 127.27.122.193:38601)
0928 16:34:41.626525 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:41.626565 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:41.627847 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:41.628337 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:41.628344 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:41.628429 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:41.629042 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:34:41.629044 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:41.629082 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:34:41.629154 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:41.629456 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:41.629467 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:41.629475 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:34:41.629673 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 16:34:41.629903 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":109,"threads_started":1}
I20260928 16:34:41.631976 503 heartbeater.cc:347] Connected to a master server at 127.27.122.254:36227
I20260928 16:34:41.632213 503 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:41.632596 503 heartbeater.cc:511] Master 127.27.122.254:36227 requested a full tablet report, sending...
I20260928 16:34:41.634127 314 ts_manager.cc:194] Registered new tserver with Master: f7fce234f8cc449eb9e411b156ba3303 (127.27.122.193:41087)
W20260928 16:34:41.735842 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:41.736130 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:41.736217 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:41.740167 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:41.740247 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:41.740337 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:41.745558 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:41.746591 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:41.747944 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:41.757254 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:41.757217 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:41.757257 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:41.757364 506 server_base.cc:1034] running on GCE node
I20260928 16:34:41.757768 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:41.758376 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:41.759560 506 hybrid_clock.cc:648] HybridClock initialized: now 1790613281759543 us; error 29 us; skew 500 ppm
I20260928 16:34:41.761327 506 webserver.cc:460] Webserver started at http://127.27.122.194:36797/ using document root <none> and password file <none>
I20260928 16:34:41.761875 506 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:41.761978 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:41.762207 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:41.764151 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "8d0c348816fe45d48962afbdb086ff38"
format_stamp: "Formatted at 2026-09-28 16:34:41 on dist-test-slave-g9sh"
server_key: "7fce561678ca7621db922fe55effd316"
server_key_iv: "286c652147692aa775e71e3894794325"
server_key_version: "encryptionkey@0"
I20260928 16:34:41.764694 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "8d0c348816fe45d48962afbdb086ff38"
format_stamp: "Formatted at 2026-09-28 16:34:41 on dist-test-slave-g9sh"
server_key: "7fce561678ca7621db922fe55effd316"
server_key_iv: "286c652147692aa775e71e3894794325"
server_key_version: "encryptionkey@0"
I20260928 16:34:41.768244 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:34:41.770645 520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:41.771811 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 16:34:41.771951 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "8d0c348816fe45d48962afbdb086ff38"
format_stamp: "Formatted at 2026-09-28 16:34:41 on dist-test-slave-g9sh"
server_key: "7fce561678ca7621db922fe55effd316"
server_key_iv: "286c652147692aa775e71e3894794325"
server_key_version: "encryptionkey@0"
I20260928 16:34:41.772061 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:41.776461 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:41.777097 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:41.777271 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:41.777791 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:41.778663 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:41.778733 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:41.778800 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:41.778852 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:41.787467 506 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:37685
I20260928 16:34:41.787485 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:37685 every 8 connection(s)
I20260928 16:34:41.788339 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:41.792629 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 506
I20260928 16:34:41.792764 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:34:41.793109 28139 external_mini_cluster.cc:1428] Setting key 55e47c3c52e05c0bf1b805cf74d5f93c
I20260928 16:34:41.794535 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.790178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:35537 (local address 127.27.122.254:36227)
0928 16:34:41.790321 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:41.790325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:41.791346 (+ 1021us) server_negotiation.cc:407] Connection header received
0928 16:34:41.792677 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:41.792681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:41.792731 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:41.792847 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:41.794069 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:41.794072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:41.794075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:41.794086 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:41.794124 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:41.794135 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:41.794224 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:41.794352 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 16:34:41.794396 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:34:41.795207 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36227
--builtin_ntp_servers=127.27.122.212:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 16:34:41.795470 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.790564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:36227 (local address 127.27.122.194:35537)
0928 16:34:41.791202 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:41.791241 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:41.792423 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:41.793040 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:41.793047 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:41.793135 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:41.793788 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:34:41.793790 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:41.793843 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:34:41.793925 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:41.794222 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:41.794229 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:41.794234 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:34:41.794361 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 16:34:41.794636 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":105,"threads_started":1}
I20260928 16:34:41.796582 635 heartbeater.cc:347] Connected to a master server at 127.27.122.254:36227
I20260928 16:34:41.796798 635 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:41.797246 635 heartbeater.cc:511] Master 127.27.122.254:36227 requested a full tablet report, sending...
I20260928 16:34:41.798250 314 ts_manager.cc:194] Registered new tserver with Master: 8d0c348816fe45d48962afbdb086ff38 (127.27.122.194:37685)
W20260928 16:34:41.907294 638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:41.907536 638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:41.907573 638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:41.911757 638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:41.911806 638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:41.911895 638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:41.916642 638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:36227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:41.917691 638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:41.919135 638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:41.928797 643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:41.928797 644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:41.928813 646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:41.929219 638 server_base.cc:1034] running on GCE node
I20260928 16:34:41.929621 638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:41.930275 638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:41.931427 638 hybrid_clock.cc:648] HybridClock initialized: now 1790613281931392 us; error 47 us; skew 500 ppm
I20260928 16:34:41.933288 638 webserver.cc:460] Webserver started at http://127.27.122.195:41721/ using document root <none> and password file <none>
I20260928 16:34:41.933815 638 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:41.933873 638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:41.934033 638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:41.936014 638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "b6c6921052e848318aa674948fe4db30"
format_stamp: "Formatted at 2026-09-28 16:34:41 on dist-test-slave-g9sh"
server_key: "3adc431b335fe2151f68d6fe64e3654a"
server_key_iv: "77669f3261e506feb5192e223571ee2a"
server_key_version: "encryptionkey@0"
I20260928 16:34:41.936542 638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "b6c6921052e848318aa674948fe4db30"
format_stamp: "Formatted at 2026-09-28 16:34:41 on dist-test-slave-g9sh"
server_key: "3adc431b335fe2151f68d6fe64e3654a"
server_key_iv: "77669f3261e506feb5192e223571ee2a"
server_key_version: "encryptionkey@0"
I20260928 16:34:41.940487 638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:34:41.942876 652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:41.944047 638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:41.944154 638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "b6c6921052e848318aa674948fe4db30"
format_stamp: "Formatted at 2026-09-28 16:34:41 on dist-test-slave-g9sh"
server_key: "3adc431b335fe2151f68d6fe64e3654a"
server_key_iv: "77669f3261e506feb5192e223571ee2a"
server_key_version: "encryptionkey@0"
I20260928 16:34:41.944238 638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:41.948541 638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:41.949138 638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:41.949270 638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:41.949719 638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:41.950531 638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:41.950575 638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:41.950614 638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:41.950636 638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:41.958988 638 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:46099
I20260928 16:34:41.959010 766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:46099 every 8 connection(s)
I20260928 16:34:41.959872 638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:34:41.961493 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 638
I20260928 16:34:41.961620 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:34:41.961905 28139 external_mini_cluster.cc:1428] Setting key 10f669311975c83f3542fcd44ec94f60
I20260928 16:34:41.966025 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.961655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:32879 (local address 127.27.122.254:36227)
0928 16:34:41.961829 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:41.961834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:41.963101 (+ 1267us) server_negotiation.cc:407] Connection header received
0928 16:34:41.964592 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:41.964595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:41.964634 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:41.964701 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:41.965704 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:41.965707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:41.965709 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:41.965721 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:41.965741 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:41.965749 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:41.965816 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:41.965896 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 16:34:41.965930 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 16:34:41.966856 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.962111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:36227 (local address 127.27.122.195:32879)
0928 16:34:41.962941 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:41.962992 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:41.964409 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:41.964822 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:41.964829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:41.964962 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:41.965462 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:34:41.965464 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:41.965503 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:34:41.965567 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:41.965805 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:41.965809 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:41.965813 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:34:41.965930 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 16:34:41.966162 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":141,"threads_started":1}
I20260928 16:34:41.967892 767 heartbeater.cc:347] Connected to a master server at 127.27.122.254:36227
I20260928 16:34:41.968117 767 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:41.968497 767 heartbeater.cc:511] Master 127.27.122.254:36227 requested a full tablet report, sending...
I20260928 16:34:41.969488 314 ts_manager.cc:194] Registered new tserver with Master: b6c6921052e848318aa674948fe4db30 (127.27.122.195:46099)
I20260928 16:34:41.975674 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:34:41.978816 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.977526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.27.122.254:36227)
0928 16:34:41.977641 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:41.977644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:41.977800 (+ 156us) server_negotiation.cc:407] Connection header received
0928 16:34:41.977892 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:41.977894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:41.977921 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:41.977982 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:41.978544 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:41.978547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:41.978549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:41.978559 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:41.978578 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:41.978586 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:41.978651 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:41.978698 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 16:34:41.978728 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 16:34:41.981355 314 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:34:41.984241 314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:41.995664 779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.992326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.27.122.194:37685)
0928 16:34:41.992919 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:41.992936 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:41.993460 (+ 524us) server_negotiation.cc:407] Connection header received
0928 16:34:41.993523 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:41.993529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:41.993746 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:41.993883 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:41.995019 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:41.995028 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:41.995040 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:41.995054 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:41.995096 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:41.995136 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:41.995238 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:41.995413 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 16:34:41.995494 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":131,"threads_started":1}
I20260928 16:34:41.995714 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.993057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:46099 (local address 127.0.0.1:57944)
0928 16:34:41.993905 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:41.993919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:41.993997 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:41.994469 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:41.994472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:41.994486 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:41.995044 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:34:41.995046 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:41.995055 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:34:41.995087 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:41.995465 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:41.995469 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:41.995472 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 16:34:41.995542 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 16:34:41.995573 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":94,"threads_started":1}
I20260928 16:34:41.996104 778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.992372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:41087 (local address 127.0.0.1:56820)
0928 16:34:41.993194 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:41.993244 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:41.993354 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:41.994646 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:41.994649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:41.994670 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:41.995296 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:34:41.995297 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:41.995305 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:34:41.995324 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:41.995861 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:41.995865 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:41.995868 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 16:34:41.995931 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 16:34:41.995968 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"spinlock_wait_cycles":11008,"thread_start_us":102,"threads_started":1}
I20260928 16:34:41.996114 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.993466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.27.122.195:46099)
0928 16:34:41.993833 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:41.993844 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:41.994006 (+ 162us) server_negotiation.cc:407] Connection header received
0928 16:34:41.994065 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:41.994070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:41.994233 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:41.994361 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:41.995222 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:41.995228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:41.995238 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:41.995251 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:41.995294 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:41.995329 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:41.995425 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:41.995870 (+ 445us) server_negotiation.cc:299] Negotiation successful
0928 16:34:41.995941 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":141,"threads_started":1}
I20260928 16:34:41.996610 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.992531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.27.122.193:41087)
0928 16:34:41.992982 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:41.992995 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:41.993496 (+ 501us) server_negotiation.cc:407] Connection header received
0928 16:34:41.993550 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:41.993554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:41.993753 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:41.993869 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:41.995488 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:41.995494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:41.995504 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:41.995515 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:41.995555 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:41.995595 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:41.996330 (+ 735us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:41.996381 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 16:34:41.996450 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":112,"threads_started":1}
I20260928 16:34:41.996306 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:41.992169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:37685 (local address 127.0.0.1:53258)
0928 16:34:41.993194 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:41.993242 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:41.993355 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:41.994128 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:41.994133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:41.994207 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:41.994790 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:34:41.994794 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:41.994817 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:34:41.994885 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:41.995236 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:41.995245 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:41.995251 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:34:41.995381 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 16:34:41.996181 (+ 800us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"spinlock_wait_cycles":2176,"thread_start_us":64,"threads_started":1}
I20260928 16:34:41.999233 437 tablet_service.cc:1467] Processing CreateTablet for tablet ebcc6dc8c53a4204a7485980949d1296 (DEFAULT_TABLE table=test-table [id=05d5f224c8a84552b2be5ed998cf8357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:41.999285 568 tablet_service.cc:1467] Processing CreateTablet for tablet ebcc6dc8c53a4204a7485980949d1296 (DEFAULT_TABLE table=test-table [id=05d5f224c8a84552b2be5ed998cf8357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:41.999258 700 tablet_service.cc:1467] Processing CreateTablet for tablet ebcc6dc8c53a4204a7485980949d1296 (DEFAULT_TABLE table=test-table [id=05d5f224c8a84552b2be5ed998cf8357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:42.000200 437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcc6dc8c53a4204a7485980949d1296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:42.000252 700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcc6dc8c53a4204a7485980949d1296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:42.000258 568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebcc6dc8c53a4204a7485980949d1296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:42.006474 784 tablet_bootstrap.cc:492] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303: Bootstrap starting.
I20260928 16:34:42.007181 785 tablet_bootstrap.cc:492] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30: Bootstrap starting.
I20260928 16:34:42.008066 786 tablet_bootstrap.cc:492] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38: Bootstrap starting.
I20260928 16:34:42.009063 784 tablet_bootstrap.cc:654] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:42.009315 785 tablet_bootstrap.cc:654] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:42.009994 784 log.cc:824] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:42.010020 785 log.cc:824] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:42.010061 786 tablet_bootstrap.cc:654] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:42.010978 786 log.cc:824] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:42.011956 784 tablet_bootstrap.cc:492] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303: No bootstrap required, opened a new log
I20260928 16:34:42.012154 784 ts_tablet_manager.cc:1397] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260928 16:34:42.012354 785 tablet_bootstrap.cc:492] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30: No bootstrap required, opened a new log
I20260928 16:34:42.012531 785 ts_tablet_manager.cc:1397] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 16:34:42.012825 786 tablet_bootstrap.cc:492] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38: No bootstrap required, opened a new log
I20260928 16:34:42.013037 786 ts_tablet_manager.cc:1397] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:34:42.016503 786 raft_consensus.cc:348] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d0c348816fe45d48962afbdb086ff38" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 37685 } } peers { permanent_uuid: "f7fce234f8cc449eb9e411b156ba3303" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41087 } } peers { permanent_uuid: "b6c6921052e848318aa674948fe4db30" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46099 } }
I20260928 16:34:42.016736 786 raft_consensus.cc:374] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:42.016788 786 raft_consensus.cc:729] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0c348816fe45d48962afbdb086ff38, State: Initialized, Role: FOLLOWER
I20260928 16:34:42.016764 785 raft_consensus.cc:348] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d0c348816fe45d48962afbdb086ff38" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 37685 } } peers { permanent_uuid: "f7fce234f8cc449eb9e411b156ba3303" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41087 } } peers { permanent_uuid: "b6c6921052e848318aa674948fe4db30" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46099 } }
I20260928 16:34:42.016764 784 raft_consensus.cc:348] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d0c348816fe45d48962afbdb086ff38" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 37685 } } peers { permanent_uuid: "f7fce234f8cc449eb9e411b156ba3303" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41087 } } peers { permanent_uuid: "b6c6921052e848318aa674948fe4db30" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46099 } }
I20260928 16:34:42.017045 785 raft_consensus.cc:374] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:42.017045 784 raft_consensus.cc:374] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:42.017098 784 raft_consensus.cc:729] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fce234f8cc449eb9e411b156ba3303, State: Initialized, Role: FOLLOWER
I20260928 16:34:42.017112 785 raft_consensus.cc:729] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6c6921052e848318aa674948fe4db30, State: Initialized, Role: FOLLOWER
I20260928 16:34:42.017320 786 consensus_queue.cc:260] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d0c348816fe45d48962afbdb086ff38" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 37685 } } peers { permanent_uuid: "f7fce234f8cc449eb9e411b156ba3303" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41087 } } peers { permanent_uuid: "b6c6921052e848318aa674948fe4db30" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46099 } }
I20260928 16:34:42.017534 785 consensus_queue.cc:260] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d0c348816fe45d48962afbdb086ff38" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 37685 } } peers { permanent_uuid: "f7fce234f8cc449eb9e411b156ba3303" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41087 } } peers { permanent_uuid: "b6c6921052e848318aa674948fe4db30" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46099 } }
I20260928 16:34:42.017534 784 consensus_queue.cc:260] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d0c348816fe45d48962afbdb086ff38" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 37685 } } peers { permanent_uuid: "f7fce234f8cc449eb9e411b156ba3303" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41087 } } peers { permanent_uuid: "b6c6921052e848318aa674948fe4db30" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46099 } }
I20260928 16:34:42.018019 635 heartbeater.cc:503] Master 127.27.122.254:36227 was elected leader, sending a full tablet report...
I20260928 16:34:42.018491 784 ts_tablet_manager.cc:1428] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260928 16:34:42.018491 785 ts_tablet_manager.cc:1428] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 16:34:42.018832 767 heartbeater.cc:503] Master 127.27.122.254:36227 was elected leader, sending a full tablet report...
I20260928 16:34:42.019012 503 heartbeater.cc:503] Master 127.27.122.254:36227 was elected leader, sending a full tablet report...
I20260928 16:34:42.019567 786 ts_tablet_manager.cc:1428] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260928 16:34:42.039366 636 tablet.cc:2369] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:34:42.124640 504 tablet.cc:2369] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:34:42.210877 768 tablet.cc:2369] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:42.269768 792 raft_consensus.cc:482] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:42.269997 792 raft_consensus.cc:504] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d0c348816fe45d48962afbdb086ff38" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 37685 } } peers { permanent_uuid: "f7fce234f8cc449eb9e411b156ba3303" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41087 } } peers { permanent_uuid: "b6c6921052e848318aa674948fe4db30" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46099 } }
I20260928 16:34:42.271194 792 leader_election.cc:288] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d0c348816fe45d48962afbdb086ff38 (127.27.122.194:37685), f7fce234f8cc449eb9e411b156ba3303 (127.27.122.193:41087)
I20260928 16:34:42.273320 779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:42.271729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:54905 (local address 127.27.122.194:37685)
0928 16:34:42.271866 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:42.271869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:42.271881 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:34:42.271923 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:42.271925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:42.271957 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:42.272060 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:42.272864 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:42.272872 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:42.272874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:42.272884 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:42.272942 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:42.272953 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:42.273047 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:42.273143 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 16:34:42.273188 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:34:42.273320 769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:42.271488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:37685 (local address 127.27.122.195:54905)
0928 16:34:42.271595 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:42.271612 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:42.271699 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:42.272099 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:42.272103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:42.272122 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:42.272707 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:34:42.272708 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:42.272718 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:34:42.272742 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:42.273044 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:42.273047 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:42.273051 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:34:42.273145 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 16:34:42.273192 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 16:34:42.273506 781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:42.271617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:41053 (local address 127.27.122.193:41087)
0928 16:34:42.271740 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:42.271744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:42.272211 (+ 467us) server_negotiation.cc:407] Connection header received
0928 16:34:42.272275 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:42.272280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:42.272328 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:42.272399 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:42.273136 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:42.273139 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:42.273142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:42.273155 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:42.273179 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:42.273187 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:42.273284 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:42.273347 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 16:34:42.273387 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 16:34:42.273524 793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:42.271488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:41087 (local address 127.27.122.195:41053)
0928 16:34:42.272113 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:42.272124 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:42.272201 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:42.272447 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:42.272452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:42.272467 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:42.272951 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:34:42.272953 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:42.272969 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:34:42.272994 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:42.273268 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:42.273271 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:42.273274 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 16:34:42.273352 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 16:34:42.273387 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":278,"thread_start_us":138,"threads_started":1}
I20260928 16:34:42.273995 589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcc6dc8c53a4204a7485980949d1296" candidate_uuid: "b6c6921052e848318aa674948fe4db30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d0c348816fe45d48962afbdb086ff38" is_pre_election: true
I20260928 16:34:42.274187 457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcc6dc8c53a4204a7485980949d1296" candidate_uuid: "b6c6921052e848318aa674948fe4db30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fce234f8cc449eb9e411b156ba3303" is_pre_election: true
I20260928 16:34:42.274297 589 raft_consensus.cc:2454] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c6921052e848318aa674948fe4db30 in term 0.
I20260928 16:34:42.274492 457 raft_consensus.cc:2454] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b6c6921052e848318aa674948fe4db30 in term 0.
I20260928 16:34:42.274781 656 leader_election.cc:302] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d0c348816fe45d48962afbdb086ff38, b6c6921052e848318aa674948fe4db30; no voters:
I20260928 16:34:42.275065 792 raft_consensus.cc:2781] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:42.275148 792 raft_consensus.cc:482] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:42.275198 792 raft_consensus.cc:3037] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:42.276474 792 raft_consensus.cc:504] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d0c348816fe45d48962afbdb086ff38" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 37685 } } peers { permanent_uuid: "f7fce234f8cc449eb9e411b156ba3303" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41087 } } peers { permanent_uuid: "b6c6921052e848318aa674948fe4db30" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46099 } }
I20260928 16:34:42.276801 792 leader_election.cc:288] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [CANDIDATE]: Term 1 election: Requested vote from peers 8d0c348816fe45d48962afbdb086ff38 (127.27.122.194:37685), f7fce234f8cc449eb9e411b156ba3303 (127.27.122.193:41087)
I20260928 16:34:42.277168 457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcc6dc8c53a4204a7485980949d1296" candidate_uuid: "b6c6921052e848318aa674948fe4db30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7fce234f8cc449eb9e411b156ba3303"
I20260928 16:34:42.277169 589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebcc6dc8c53a4204a7485980949d1296" candidate_uuid: "b6c6921052e848318aa674948fe4db30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d0c348816fe45d48962afbdb086ff38"
I20260928 16:34:42.277297 457 raft_consensus.cc:3037] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:42.277309 589 raft_consensus.cc:3037] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:42.278324 589 raft_consensus.cc:2454] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c6921052e848318aa674948fe4db30 in term 1.
I20260928 16:34:42.278338 457 raft_consensus.cc:2454] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b6c6921052e848318aa674948fe4db30 in term 1.
I20260928 16:34:42.278614 656 leader_election.cc:302] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d0c348816fe45d48962afbdb086ff38, b6c6921052e848318aa674948fe4db30; no voters:
I20260928 16:34:42.278803 792 raft_consensus.cc:2781] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:42.279045 792 raft_consensus.cc:686] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [term 1 LEADER]: Becoming Leader. State: Replica: b6c6921052e848318aa674948fe4db30, State: Running, Role: LEADER
I20260928 16:34:42.279387 792 consensus_queue.cc:237] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d0c348816fe45d48962afbdb086ff38" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 37685 } } peers { permanent_uuid: "f7fce234f8cc449eb9e411b156ba3303" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41087 } } peers { permanent_uuid: "b6c6921052e848318aa674948fe4db30" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46099 } }
I20260928 16:34:42.282708 314 catalog_manager.cc:5495] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 reported cstate change: term changed from 0 to 1, leader changed from <none> to b6c6921052e848318aa674948fe4db30 (127.27.122.195). New cstate: current_term: 1 leader_uuid: "b6c6921052e848318aa674948fe4db30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d0c348816fe45d48962afbdb086ff38" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 37685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7fce234f8cc449eb9e411b156ba3303" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b6c6921052e848318aa674948fe4db30" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46099 } health_report { overall_health: HEALTHY } } }
I20260928 16:34:42.358050 783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:42.356671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.27.122.195:46099)
0928 16:34:42.356794 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:42.356797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:42.356810 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:34:42.356979 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:42.356981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:42.357046 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:42.357132 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:42.357747 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:42.357750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:42.357752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:42.357763 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:42.357782 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:42.357790 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:42.357862 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:42.357917 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 16:34:42.357949 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:34:42.363924 589 raft_consensus.cc:1261] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38 [term 1 FOLLOWER]: Refusing update from remote peer b6c6921052e848318aa674948fe4db30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:42.363946 457 raft_consensus.cc:1261] T ebcc6dc8c53a4204a7485980949d1296 P f7fce234f8cc449eb9e411b156ba3303 [term 1 FOLLOWER]: Refusing update from remote peer b6c6921052e848318aa674948fe4db30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:42.364593 792 consensus_queue.cc:1035] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d0c348816fe45d48962afbdb086ff38" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 37685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:42.364806 794 consensus_queue.cc:1035] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7fce234f8cc449eb9e411b156ba3303" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:42.372922 798 mvcc.cc:205] Tried to move back new op lower bound from 7334352004555624448 to 7334352004218814464. Current Snapshot: MvccSnapshot[applied={T|T < 7334352004555624448}]
I20260928 16:34:42.374585 799 mvcc.cc:205] Tried to move back new op lower bound from 7334352004555624448 to 7334352004218814464. Current Snapshot: MvccSnapshot[applied={T|T < 7334352004555624448}]
I20260928 16:34:42.380581 314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:42.380752 314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:42.383240 314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10ea0a98604b4b30884d50b6fc783e4a: Sending DeleteTablet for 3 replicas of tablet ebcc6dc8c53a4204a7485980949d1296
I20260928 16:34:42.384004 437 tablet_service.cc:1514] Processing DeleteTablet for tablet ebcc6dc8c53a4204a7485980949d1296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:42 UTC) from {username='slave'} at 127.0.0.1:56820
I20260928 16:34:42.384004 568 tablet_service.cc:1514] Processing DeleteTablet for tablet ebcc6dc8c53a4204a7485980949d1296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:42 UTC) from {username='slave'} at 127.0.0.1:53258
I20260928 16:34:42.384137 700 tablet_service.cc:1514] Processing DeleteTablet for tablet ebcc6dc8c53a4204a7485980949d1296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:42 UTC) from {username='slave'} at 127.0.0.1:57944
I20260928 16:34:42.384544 808 tablet_replica.cc:330] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38: stopping tablet replica
I20260928 16:34:42.384655 809 tablet_replica.cc:330] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30: stopping tablet replica
I20260928 16:34:42.384979 808 raft_consensus.cc:2229] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:42.384979 809 raft_consensus.cc:2229] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:42.385301 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 374
I20260928 16:34:42.386255 808 raft_consensus.cc:2258] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:42.386502 809 raft_consensus.cc:2258] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:42.387936 808 ts_tablet_manager.cc:1905] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:42.387936 809 ts_tablet_manager.cc:1905] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:42.391201 809 ts_tablet_manager.cc:1918] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:42.391309 809 log.cc:1196] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/ebcc6dc8c53a4204a7485980949d1296
I20260928 16:34:42.391669 809 ts_tablet_manager.cc:1939] T ebcc6dc8c53a4204a7485980949d1296 P b6c6921052e848318aa674948fe4db30: Deleting consensus metadata
I20260928 16:34:42.392866 300 catalog_manager.cc:4841] TS b6c6921052e848318aa674948fe4db30 (127.27.122.195:46099): tablet ebcc6dc8c53a4204a7485980949d1296 (table test-table [id=05d5f224c8a84552b2be5ed998cf8357]) successfully deleted
I20260928 16:34:42.393532 808 ts_tablet_manager.cc:1918] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:42.393639 808 log.cc:1196] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/ebcc6dc8c53a4204a7485980949d1296
I20260928 16:34:42.393946 808 ts_tablet_manager.cc:1939] T ebcc6dc8c53a4204a7485980949d1296 P 8d0c348816fe45d48962afbdb086ff38: Deleting consensus metadata
W20260928 16:34:42.394719 32766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:41087 (error 108)
I20260928 16:34:42.395319 301 catalog_manager.cc:4841] TS 8d0c348816fe45d48962afbdb086ff38 (127.27.122.194:37685): tablet ebcc6dc8c53a4204a7485980949d1296 (table test-table [id=05d5f224c8a84552b2be5ed998cf8357]) successfully deleted
I20260928 16:34:42.395514 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 506
I20260928 16:34:42.401466 780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:42.401154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:41087 (local address 127.0.0.1:56822)
0928 16:34:42.401332 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:42.401392 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:41087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260928 16:34:42.401630 32766 catalog_manager.cc:4584] TS f7fce234f8cc449eb9e411b156ba3303 (127.27.122.193:41087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ebcc6dc8c53a4204a7485980949d1296: Network error: Client connection negotiation failed: client connection to 127.27.122.193:41087: connect: Connection refused (error 111)
I20260928 16:34:42.403087 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 638
I20260928 16:34:42.408404 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 32751
2026-09-28T16:34:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1198 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 16:34:42.428723 28139 mini_oidc.cc:152] Starting JWKS server
I20260928 16:34:42.430428 28139 webserver.cc:460] Webserver started at https://127.0.0.1:36427/ using document root <none> and password file <none>
I20260928 16:34:42.430604 28139 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:34:42.430884 28139 webserver.cc:460] Webserver started at http://127.0.0.1:45983/ using document root <none> and password file <none>
2026-09-28T16:34:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:42Z Disabled control of system clock
I20260928 16:34:42.439680 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:36043
--webserver_interface=127.27.122.254
--webserver_port=0
--builtin_ntp_servers=127.27.122.212:43619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:36043
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36427/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/testchainca.cert with env {}
W20260928 16:34:42.548889 816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:42.549167 816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:42.549206 816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:42.553056 816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:42.553097 816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:42.553112 816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:42.553128 816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:42.553144 816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:34:42.553159 816 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:34:42.553174 816 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36427/jwks.json
I20260928 16:34:42.558252 816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:36043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:36043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36427/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:42.559190 816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:42.560111 816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:34:42.568403 816 server_base.cc:1034] running on GCE node
W20260928 16:34:42.568300 821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:42.568280 824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:42.568281 822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:42.569170 816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:42.570158 816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:42.571321 816 hybrid_clock.cc:648] HybridClock initialized: now 1790613282571292 us; error 42 us; skew 500 ppm
I20260928 16:34:42.573294 816 webserver.cc:460] Webserver started at http://127.27.122.254:39895/ using document root <none> and password file <none>
I20260928 16:34:42.573810 816 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:42.573869 816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:42.574026 816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:42.575977 816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "5e89d91d904649deb84dbcebd04c4659"
format_stamp: "Formatted at 2026-09-28 16:34:42 on dist-test-slave-g9sh"
server_key: "94463fc8e7fceeabf715e402b6da8c3e"
server_key_iv: "e07310ef33c1be169d155df48c1ff3b1"
server_key_version: "encryptionkey@0"
I20260928 16:34:42.576488 816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "5e89d91d904649deb84dbcebd04c4659"
format_stamp: "Formatted at 2026-09-28 16:34:42 on dist-test-slave-g9sh"
server_key: "94463fc8e7fceeabf715e402b6da8c3e"
server_key_iv: "e07310ef33c1be169d155df48c1ff3b1"
server_key_version: "encryptionkey@0"
I20260928 16:34:42.579974 816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:34:42.582347 830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:42.583524 816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:34:42.583622 816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "5e89d91d904649deb84dbcebd04c4659"
format_stamp: "Formatted at 2026-09-28 16:34:42 on dist-test-slave-g9sh"
server_key: "94463fc8e7fceeabf715e402b6da8c3e"
server_key_iv: "e07310ef33c1be169d155df48c1ff3b1"
server_key_version: "encryptionkey@0"
I20260928 16:34:42.583704 816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:42.598635 816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:42.599272 816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:42.599419 816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:42.607280 816 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:36043
I20260928 16:34:42.607286 883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:36043 every 8 connection(s)
I20260928 16:34:42.608229 816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:42.611334 884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:42.615046 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 816
I20260928 16:34:42.615191 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:42.615491 28139 external_mini_cluster.cc:1428] Setting key be6c15e2cdd6c481dd3fce289cf0a614
I20260928 16:34:42.617465 884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659: Bootstrap starting.
I20260928 16:34:42.620196 884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:42.621120 884 log.cc:824] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:42.623217 884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659: No bootstrap required, opened a new log
I20260928 16:34:42.626385 887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:42.616828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.27.122.254:36043)
0928 16:34:42.617587 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:42.617608 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:42.617651 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:34:42.618484 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:42.618508 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:42.619069 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:42.619459 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:42.619934 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:42.621446 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:42.622431 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:42.623115 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:42.623289 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:42.624114 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:42.624121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:42.624136 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:42.624625 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:42.624695 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:42.624720 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:42.624821 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:42.624964 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 16:34:42.625243 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":137,"threads_started":1}
I20260928 16:34:42.627102 884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89d91d904649deb84dbcebd04c4659" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36043 } }
I20260928 16:34:42.627360 884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:42.627420 884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e89d91d904649deb84dbcebd04c4659, State: Initialized, Role: FOLLOWER
I20260928 16:34:42.627876 884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89d91d904649deb84dbcebd04c4659" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36043 } }
I20260928 16:34:42.628000 884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:42.628082 884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:42.628185 884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:42.629338 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:34:42.629298 884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89d91d904649deb84dbcebd04c4659" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36043 } }
I20260928 16:34:42.629673 884 leader_election.cc:302] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e89d91d904649deb84dbcebd04c4659; no voters:
I20260928 16:34:42.629992 884 leader_election.cc:288] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:42.630064 889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:42.630434 889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [term 1 LEADER]: Becoming Leader. State: Replica: 5e89d91d904649deb84dbcebd04c4659, State: Running, Role: LEADER
I20260928 16:34:42.630792 889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89d91d904649deb84dbcebd04c4659" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36043 } }
I20260928 16:34:42.631310 884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:42.631955 889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e89d91d904649deb84dbcebd04c4659. Latest consensus state: current_term: 1 leader_uuid: "5e89d91d904649deb84dbcebd04c4659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89d91d904649deb84dbcebd04c4659" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36043 } } }
I20260928 16:34:42.632071 889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:42.632725 890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e89d91d904649deb84dbcebd04c4659" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e89d91d904649deb84dbcebd04c4659" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36043 } } }
I20260928 16:34:42.632822 890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:42.634109 895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:42.636726 895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:42.641887 895 catalog_manager.cc:1347] Generated new cluster ID: a1e0b1b9b6d34574ad38382ab508085d
I20260928 16:34:42.641952 895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:42.658769 895 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:42.659732 895 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:42.671398 895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e89d91d904649deb84dbcebd04c4659: Generated new TSK 0
I20260928 16:34:42.672119 895 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 16:34:42.706805 914 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 16:34:42.706923 914 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 16:34:42.707293 887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:42.683372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.27.122.254:36043)
0928 16:34:42.683516 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:42.683523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:42.683762 (+ 239us) server_negotiation.cc:407] Connection header received
0928 16:34:42.684006 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:42.684009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:42.684058 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:42.684169 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:34:42.684435 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:42.685044 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:42.685937 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:42.686175 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:42.686289 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:42.686646 (+ 357us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:34:42.706866 (+ 20220us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 16:34:42.707179 (+ 313us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45682: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 16:34:42.707394 887 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45682: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 16:34:42.708735 913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:34:42.683295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:36043 (local address 127.0.0.1:45682)
0928 16:34:42.683678 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:42.683690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:42.683838 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:42.684189 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:42.684192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:42.684207 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:34:42.684325 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:42.684330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:42.685162 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:42.685165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:42.685802 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:42.685810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:42.686296 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:42.686300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:42.686402 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:42.686444 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:34:42.707183 (+ 20739us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 16:34:42.707415 (+ 232us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.122.254:36043: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":114,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.122.254:36043: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 16:34:42.709515 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 816
2026-09-28T16:34:42Z chronyd exiting
I20260928 16:34:42.746495 28139 test_util.cc:175] -----------------------------------------------
I20260928 16:34:42.746601 28139 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790613255639823-28139-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (319 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 16:34:42.747660 28139 mini_oidc.cc:152] Starting JWKS server
I20260928 16:34:42.748512 28139 webserver.cc:460] Webserver started at https://127.0.0.1:40319/ using document root <none> and password file <none>
I20260928 16:34:42.748646 28139 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:34:42.748972 28139 webserver.cc:460] Webserver started at http://127.0.0.1:33121/ using document root <none> and password file <none>
2026-09-28T16:34:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:42Z Disabled control of system clock
I20260928 16:34:42.757390 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:36863
--webserver_interface=127.27.122.254
--webserver_port=0
--builtin_ntp_servers=127.27.122.212:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:36863
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40319/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/testchainca.cert with env {}
W20260928 16:34:42.864873 920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:42.865180 920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:42.865218 920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:42.869115 920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:42.869161 920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:42.869176 920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:42.869192 920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:42.869207 920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:34:42.869223 920 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:34:42.869236 920 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40319/jwks.json
I20260928 16:34:42.874253 920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:36863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:36863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40319/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:42.875150 920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:42.876050 920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:42.884439 926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:42.884516 920 server_base.cc:1034] running on GCE node
W20260928 16:34:42.884479 925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:42.884639 928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:42.885274 920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:42.886266 920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:42.887426 920 hybrid_clock.cc:648] HybridClock initialized: now 1790613282887395 us; error 43 us; skew 500 ppm
I20260928 16:34:42.889249 920 webserver.cc:460] Webserver started at http://127.27.122.254:40679/ using document root <none> and password file <none>
I20260928 16:34:42.889807 920 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:42.889869 920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:42.890123 920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:42.892180 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "e53b76a8ff784df09cbdc56a77cd44ac"
format_stamp: "Formatted at 2026-09-28 16:34:42 on dist-test-slave-g9sh"
server_key: "180d5f2cbbe5cd68fd08bee558c64e07"
server_key_iv: "f7411f3418812389501f4b1d61c0ede3"
server_key_version: "encryptionkey@0"
I20260928 16:34:42.892776 920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "e53b76a8ff784df09cbdc56a77cd44ac"
format_stamp: "Formatted at 2026-09-28 16:34:42 on dist-test-slave-g9sh"
server_key: "180d5f2cbbe5cd68fd08bee558c64e07"
server_key_iv: "f7411f3418812389501f4b1d61c0ede3"
server_key_version: "encryptionkey@0"
I20260928 16:34:42.896394 920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:34:42.898728 934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:42.899904 920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:42.900044 920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "e53b76a8ff784df09cbdc56a77cd44ac"
format_stamp: "Formatted at 2026-09-28 16:34:42 on dist-test-slave-g9sh"
server_key: "180d5f2cbbe5cd68fd08bee558c64e07"
server_key_iv: "f7411f3418812389501f4b1d61c0ede3"
server_key_version: "encryptionkey@0"
I20260928 16:34:42.900164 920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:42.915865 920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:42.916507 920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:42.916702 920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:42.923758 920 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:36863
I20260928 16:34:42.923763 988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:36863 every 8 connection(s)
I20260928 16:34:42.924664 920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:42.927786 989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:42.932359 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 920
I20260928 16:34:42.932507 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:42.932767 28139 external_mini_cluster.cc:1428] Setting key 3227750691cfe742d72294cf72ec642d
I20260928 16:34:42.933341 989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac: Bootstrap starting.
I20260928 16:34:42.935595 989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:42.936480 989 log.cc:824] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:42.938570 989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac: No bootstrap required, opened a new log
I20260928 16:34:42.941974 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:42.934074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.27.122.254:36863)
0928 16:34:42.934653 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:42.934668 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:42.934703 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:34:42.935321 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:42.935337 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:42.935765 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:42.936038 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:42.936383 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:42.937604 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:42.938806 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:42.939277 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:42.939399 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:42.940252 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:42.940257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:42.940269 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:42.940641 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:42.940694 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:42.940713 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:42.940790 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:42.941013 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 16:34:42.941300 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":123,"threads_started":1}
I20260928 16:34:42.942560 989 raft_consensus.cc:348] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53b76a8ff784df09cbdc56a77cd44ac" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36863 } }
I20260928 16:34:42.942762 989 raft_consensus.cc:374] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:42.942821 989 raft_consensus.cc:729] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53b76a8ff784df09cbdc56a77cd44ac, State: Initialized, Role: FOLLOWER
I20260928 16:34:42.943250 989 consensus_queue.cc:260] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53b76a8ff784df09cbdc56a77cd44ac" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36863 } }
I20260928 16:34:42.943395 989 raft_consensus.cc:388] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:42.943478 989 raft_consensus.cc:482] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:42.943596 989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:42.944545 989 raft_consensus.cc:504] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53b76a8ff784df09cbdc56a77cd44ac" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36863 } }
I20260928 16:34:42.944835 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:34:42.944950 989 leader_election.cc:302] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e53b76a8ff784df09cbdc56a77cd44ac; no voters:
I20260928 16:34:42.945268 989 leader_election.cc:288] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:42.945396 994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:42.945636 994 raft_consensus.cc:686] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [term 1 LEADER]: Becoming Leader. State: Replica: e53b76a8ff784df09cbdc56a77cd44ac, State: Running, Role: LEADER
I20260928 16:34:42.945956 994 consensus_queue.cc:237] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53b76a8ff784df09cbdc56a77cd44ac" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36863 } }
I20260928 16:34:42.946504 989 sys_catalog.cc:564] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:42.947660 996 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e53b76a8ff784df09cbdc56a77cd44ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53b76a8ff784df09cbdc56a77cd44ac" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36863 } } }
I20260928 16:34:42.947671 995 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53b76a8ff784df09cbdc56a77cd44ac. Latest consensus state: current_term: 1 leader_uuid: "e53b76a8ff784df09cbdc56a77cd44ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53b76a8ff784df09cbdc56a77cd44ac" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 36863 } } }
I20260928 16:34:42.947882 995 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:42.947954 996 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:42.948446 1003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:42.951524 1003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:42.957316 1003 catalog_manager.cc:1347] Generated new cluster ID: eacbddc82275411cb9aebb027a3359ef
I20260928 16:34:42.957382 1003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:42.972714 1003 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:42.973603 1003 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:42.979866 1003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e53b76a8ff784df09cbdc56a77cd44ac: Generated new TSK 0
I20260928 16:34:42.980618 1003 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 16:34:43.008582 1019 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 16:34:43.008672 1019 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 16:34:43.009423 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:42.993732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48852 (local address 127.27.122.254:36863)
0928 16:34:42.993870 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:42.993877 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:42.994091 (+ 214us) server_negotiation.cc:407] Connection header received
0928 16:34:42.994298 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:42.994300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:42.994347 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:42.994453 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:34:42.994714 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:42.995250 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:42.996143 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:42.996375 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:42.996488 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:42.996803 (+ 315us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:34:43.009016 (+ 12213us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 16:34:43.009272 (+ 256us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 16:34:43.009498 992 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 16:34:43.009510 1018 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:34:42.993653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:36863 (local address 127.0.0.1:48852)
0928 16:34:42.994006 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:42.994018 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:42.994148 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:42.994467 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:42.994470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:42.994479 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:34:42.994605 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:42.994612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:42.995369 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:42.995373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:42.996007 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:42.996015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:42.996496 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:42.996500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:42.996590 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:42.996623 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:34:43.009181 (+ 12558us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 16:34:43.009343 (+ 162us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.122.254:36863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40319/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":107,"threads_started":1}
I20260928 16:34:43.010180 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 920
2026-09-28T16:34:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 16:34:43.049829 28139 mini_oidc.cc:152] Starting JWKS server
I20260928 16:34:43.050652 28139 webserver.cc:460] Webserver started at https://127.0.0.1:46199/ using document root <none> and password file <none>
I20260928 16:34:43.050730 28139 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 16:34:43.050935 28139 webserver.cc:460] Webserver started at http://127.0.0.1:40853/ using document root <none> and password file <none>
2026-09-28T16:34:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:43Z Disabled control of system clock
I20260928 16:34:43.059741 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:40481
--webserver_interface=127.27.122.254
--webserver_port=0
--builtin_ntp_servers=127.27.122.212:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:40481
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46199/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 16:34:43.167888 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:43.168116 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:43.168152 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:43.172065 1025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:43.172113 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:43.172147 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:43.172163 1025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:43.172178 1025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:34:43.172192 1025 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:34:43.172205 1025 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46199/jwks.json
I20260928 16:34:43.176852 1025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:40481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:40481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46199/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:43.177839 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:43.179086 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:43.187417 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:43.187502 1025 server_base.cc:1034] running on GCE node
W20260928 16:34:43.187408 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:43.187407 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:43.188227 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:43.189275 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:43.190438 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790613283190419 us; error 31 us; skew 500 ppm
I20260928 16:34:43.195933 1025 webserver.cc:460] Webserver started at http://127.27.122.254:44177/ using document root <none> and password file <none>
I20260928 16:34:43.196463 1025 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:43.196522 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:43.196700 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:43.198787 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "95af933ff9a841abb5a95b5d39349be6"
format_stamp: "Formatted at 2026-09-28 16:34:43 on dist-test-slave-g9sh"
server_key: "58d44c5744bbc84cb9a6de8da35a9005"
server_key_iv: "1d1cc83ad237c22099b9be3b0f1ab4e5"
server_key_version: "encryptionkey@0"
I20260928 16:34:43.199309 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "95af933ff9a841abb5a95b5d39349be6"
format_stamp: "Formatted at 2026-09-28 16:34:43 on dist-test-slave-g9sh"
server_key: "58d44c5744bbc84cb9a6de8da35a9005"
server_key_iv: "1d1cc83ad237c22099b9be3b0f1ab4e5"
server_key_version: "encryptionkey@0"
I20260928 16:34:43.202979 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:34:43.205320 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:43.206523 1025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:43.206634 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "95af933ff9a841abb5a95b5d39349be6"
format_stamp: "Formatted at 2026-09-28 16:34:43 on dist-test-slave-g9sh"
server_key: "58d44c5744bbc84cb9a6de8da35a9005"
server_key_iv: "1d1cc83ad237c22099b9be3b0f1ab4e5"
server_key_version: "encryptionkey@0"
I20260928 16:34:43.206720 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:43.216029 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:43.216590 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:43.216725 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:43.223412 1025 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:40481
I20260928 16:34:43.223436 1092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:40481 every 8 connection(s)
I20260928 16:34:43.224309 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:43.224925 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 1025
I20260928 16:34:43.225096 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:43.225373 28139 external_mini_cluster.cc:1428] Setting key 72fe667d6e91e266938cf4a78970ba2f
I20260928 16:34:43.227463 1093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:43.233774 1093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6: Bootstrap starting.
I20260928 16:34:43.234357 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:43.226678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45694 (local address 127.27.122.254:40481)
0928 16:34:43.227255 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:43.227269 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:43.227302 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:34:43.227973 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:43.227990 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:43.228362 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:43.228643 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:43.229020 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:43.229954 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:43.231135 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:43.231577 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:43.231689 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:43.232595 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:43.232600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:43.232612 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:43.233040 (+ 428us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:43.233095 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:43.233116 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:43.233191 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:43.233355 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 16:34:43.233646 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":133,"threads_started":1}
I20260928 16:34:43.235954 1093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:43.236766 1093 log.cc:824] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:43.236984 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:34:43.238862 1093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6: No bootstrap required, opened a new log
I20260928 16:34:43.242203 1093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95af933ff9a841abb5a95b5d39349be6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40481 } }
I20260928 16:34:43.242522 1093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:43.242614 1093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95af933ff9a841abb5a95b5d39349be6, State: Initialized, Role: FOLLOWER
I20260928 16:34:43.243031 1093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95af933ff9a841abb5a95b5d39349be6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40481 } }
I20260928 16:34:43.243183 1093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:43.243268 1093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:43.243366 1093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:43.244215 1093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95af933ff9a841abb5a95b5d39349be6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40481 } }
I20260928 16:34:43.244541 1093 leader_election.cc:302] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95af933ff9a841abb5a95b5d39349be6; no voters:
I20260928 16:34:43.244917 1093 leader_election.cc:288] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:43.244948 1098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:43.245189 1098 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [term 1 LEADER]: Becoming Leader. State: Replica: 95af933ff9a841abb5a95b5d39349be6, State: Running, Role: LEADER
I20260928 16:34:43.245477 1098 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95af933ff9a841abb5a95b5d39349be6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40481 } }
I20260928 16:34:43.246163 1093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:43.247035 1099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95af933ff9a841abb5a95b5d39349be6. Latest consensus state: current_term: 1 leader_uuid: "95af933ff9a841abb5a95b5d39349be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95af933ff9a841abb5a95b5d39349be6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40481 } } }
I20260928 16:34:43.247160 1099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:43.247498 1100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95af933ff9a841abb5a95b5d39349be6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95af933ff9a841abb5a95b5d39349be6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40481 } } }
I20260928 16:34:43.247601 1100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:43.247851 1107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:43.250845 1107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:43.256230 1107 catalog_manager.cc:1347] Generated new cluster ID: b1327acdd60d4e98a27ed0d88f54dca0
I20260928 16:34:43.256290 1107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:43.289583 1107 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:43.290850 1107 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:43.296778 1107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95af933ff9a841abb5a95b5d39349be6: Generated new TSK 0
I20260928 16:34:43.297557 1107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:34:43.312978 1096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:43.308029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.27.122.254:40481)
0928 16:34:43.308174 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:43.308181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:43.308382 (+ 201us) server_negotiation.cc:407] Connection header received
0928 16:34:43.308579 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:43.308581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:43.308627 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:43.308734 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:34:43.309019 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:43.309531 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:43.310539 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:43.310762 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:43.310874 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:43.311308 (+ 434us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:34:43.312623 (+ 1315us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46199/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 16:34:43.312837 (+ 214us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45710: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46199/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 16:34:43.313035 1096 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45710: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46199/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 16:34:43.314806 1122 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:34:43.307936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:40481 (local address 127.0.0.1:45710)
0928 16:34:43.308296 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:43.308307 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:43.308430 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:43.308735 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:43.308739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:43.308747 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:34:43.308860 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:43.308874 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:43.309654 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:43.309658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:43.310418 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:43.310427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:43.310891 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:43.310895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:43.310991 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:43.311013 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:34:43.312783 (+ 1770us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46199/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 16:34:43.314635 (+ 1852us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.122.254:40481: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46199/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":1650,"thread_start_us":117,"threads_started":1}
I20260928 16:34:43.315325 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 1025
E20260928 16:34:43.324056 1123 webserver.cc:533] Webserver: sslize failed:
E20260928 16:34:43.324138 1123 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T16:34:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T16:34:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:43Z Disabled control of system clock
I20260928 16:34:43.362741 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:42839
--webserver_interface=127.27.122.254
--webserver_port=0
--builtin_ntp_servers=127.27.122.212:34097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:42839
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/testchainca.cert with env {}
W20260928 16:34:43.471648 1127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:43.471876 1127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:43.471915 1127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:43.475823 1127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:43.475873 1127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:43.475889 1127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:43.475905 1127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:43.475921 1127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 16:34:43.475936 1127 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 16:34:43.475950 1127 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 16:34:43.481045 1127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:34097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:42839
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:42839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:43.481981 1127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:43.482944 1127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:43.491146 1132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:43.491153 1135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:43.491164 1133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:43.491204 1127 server_base.cc:1034] running on GCE node
I20260928 16:34:43.491948 1127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:43.493044 1127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:43.494236 1127 hybrid_clock.cc:648] HybridClock initialized: now 1790613283494227 us; error 30 us; skew 500 ppm
I20260928 16:34:43.495996 1127 webserver.cc:460] Webserver started at http://127.27.122.254:44663/ using document root <none> and password file <none>
I20260928 16:34:43.496579 1127 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:43.496676 1127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:43.496942 1127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:43.498955 1127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "032c1ea379854455a4e06caf04410a39"
format_stamp: "Formatted at 2026-09-28 16:34:43 on dist-test-slave-g9sh"
server_key: "61fb2a3e4e39208770bd59a6f9d39bda"
server_key_iv: "1d3b1a8c9c8265e545e1d30285e134df"
server_key_version: "encryptionkey@0"
I20260928 16:34:43.499554 1127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "032c1ea379854455a4e06caf04410a39"
format_stamp: "Formatted at 2026-09-28 16:34:43 on dist-test-slave-g9sh"
server_key: "61fb2a3e4e39208770bd59a6f9d39bda"
server_key_iv: "1d3b1a8c9c8265e545e1d30285e134df"
server_key_version: "encryptionkey@0"
I20260928 16:34:43.503146 1127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:34:43.505527 1141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:43.506687 1127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260928 16:34:43.506839 1127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "032c1ea379854455a4e06caf04410a39"
format_stamp: "Formatted at 2026-09-28 16:34:43 on dist-test-slave-g9sh"
server_key: "61fb2a3e4e39208770bd59a6f9d39bda"
server_key_iv: "1d3b1a8c9c8265e545e1d30285e134df"
server_key_version: "encryptionkey@0"
I20260928 16:34:43.506951 1127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:43.525969 1127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:43.526620 1127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:43.526819 1127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:43.533942 1127 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:42839
I20260928 16:34:43.533947 1193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:42839 every 8 connection(s)
I20260928 16:34:43.534886 1127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:43.537818 1194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:43.538041 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 1127
I20260928 16:34:43.538167 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:43.538476 28139 external_mini_cluster.cc:1428] Setting key 4bd1001464130aad5a97738cd3f9b1f0
I20260928 16:34:43.544651 1194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39: Bootstrap starting.
I20260928 16:34:43.547230 1194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:43.547658 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:43.539773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.27.122.254:42839)
0928 16:34:43.540374 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:43.540389 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:43.540427 (+ 38us) server_negotiation.cc:407] Connection header received
0928 16:34:43.541124 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:43.541142 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:43.541541 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:43.541816 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:43.542153 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:43.543081 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:43.544288 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:43.544835 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:43.545014 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:43.545836 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:43.545840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:43.545852 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:34:43.546235 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:43.546290 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:34:43.546311 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:43.546389 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:43.546550 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 16:34:43.546847 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":127,"threads_started":1}
I20260928 16:34:43.548026 1194 log.cc:824] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:43.550149 1194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39: No bootstrap required, opened a new log
I20260928 16:34:43.550498 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:34:43.553910 1194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c1ea379854455a4e06caf04410a39" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42839 } }
I20260928 16:34:43.554163 1194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:43.554256 1194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 032c1ea379854455a4e06caf04410a39, State: Initialized, Role: FOLLOWER
I20260928 16:34:43.554682 1194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c1ea379854455a4e06caf04410a39" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42839 } }
I20260928 16:34:43.554833 1194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:43.554924 1194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:43.555049 1194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:43.556177 1194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c1ea379854455a4e06caf04410a39" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42839 } }
I20260928 16:34:43.556523 1194 leader_election.cc:302] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 032c1ea379854455a4e06caf04410a39; no voters:
I20260928 16:34:43.556836 1194 leader_election.cc:288] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:43.557001 1199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:43.557224 1199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [term 1 LEADER]: Becoming Leader. State: Replica: 032c1ea379854455a4e06caf04410a39, State: Running, Role: LEADER
I20260928 16:34:43.557544 1199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c1ea379854455a4e06caf04410a39" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42839 } }
I20260928 16:34:43.558125 1194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:43.559303 1201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 032c1ea379854455a4e06caf04410a39. Latest consensus state: current_term: 1 leader_uuid: "032c1ea379854455a4e06caf04410a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c1ea379854455a4e06caf04410a39" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42839 } } }
I20260928 16:34:43.559448 1201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:43.559876 1208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:43.559762 1200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "032c1ea379854455a4e06caf04410a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "032c1ea379854455a4e06caf04410a39" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42839 } } }
I20260928 16:34:43.560527 1200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:43.562914 1208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:43.568567 1208 catalog_manager.cc:1347] Generated new cluster ID: 37302af35a134aa7b8855d5a13578dd1
I20260928 16:34:43.568645 1208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:43.573451 1208 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:43.574599 1208 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:43.580516 1208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 032c1ea379854455a4e06caf04410a39: Generated new TSK 0
I20260928 16:34:43.581293 1208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:34:43.667436 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:43.589406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.27.122.254:42839)
0928 16:34:43.589540 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:43.589545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:43.589783 (+ 238us) server_negotiation.cc:407] Connection header received
0928 16:34:43.589959 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:43.589962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:43.590007 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:43.590101 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0928 16:34:43.590358 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:43.590863 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:43.592110 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:43.592316 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:43.592387 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:43.592725 (+ 338us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 16:34:43.667030 (+ 74305us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 16:34:43.667315 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33818: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 16:34:43.667539 1197 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33818: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 16:34:43.668061 1223 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:34:43.589344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42839 (local address 127.0.0.1:33818)
0928 16:34:43.589677 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:43.589693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:43.589839 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:43.590126 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:43.590129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:43.590138 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 16:34:43.590255 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:43.590261 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:43.590990 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:43.590994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:43.591994 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:43.592004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:43.592436 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:43.592440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:43.592539 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:43.592566 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 16:34:43.667298 (+ 74732us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 16:34:43.667529 (+ 231us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.122.254:42839: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":120,"threads_started":1}
I20260928 16:34:43.668823 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 1127
2026-09-28T16:34:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (333 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (120 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:43 dist-test-slave-g9sh krb5kdc[1235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:43 dist-test-slave-g9sh krb5kdc[1235](info): set up 2 sockets
Sep 28 16:34:43 dist-test-slave-g9sh krb5kdc[1235](info): commencing operation
krb5kdc: starting...
W20260928 16:34:44.947793 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:34:45.065879 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:33927
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:33927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:45.172392 1249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:45.172626 1249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:45.172663 1249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:45.176617 1249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:45.176667 1249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:45.176687 1249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:45.176702 1249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:45.176717 1249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:45.181818 1249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:33927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:33927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:45.182741 1249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:45.183697 1249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:45.192200 1254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:45.192209 1257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:45.192250 1249 server_base.cc:1034] running on GCE node
W20260928 16:34:45.192209 1255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:45.193077 1249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:45.194114 1249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:45.195277 1249 hybrid_clock.cc:648] HybridClock initialized: now 1790613285195259 us; error 29 us; skew 500 ppm
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:45.197929 1249 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:34:45.198743 1249 webserver.cc:460] Webserver started at http://127.27.122.254:33995/ using document root <none> and password file <none>
I20260928 16:34:45.199287 1249 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:45.199347 1249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:45.199575 1249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:45.201637 1249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "3dd875f68ab248caa944d9d288b91c82"
format_stamp: "Formatted at 2026-09-28 16:34:45 on dist-test-slave-g9sh"
server_key: "efe120510958457e3e32597090339baf"
server_key_iv: "58b967d09512230a00020e6b56fbaf54"
server_key_version: "encryptionkey@0"
I20260928 16:34:45.202189 1249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "3dd875f68ab248caa944d9d288b91c82"
format_stamp: "Formatted at 2026-09-28 16:34:45 on dist-test-slave-g9sh"
server_key: "efe120510958457e3e32597090339baf"
server_key_iv: "58b967d09512230a00020e6b56fbaf54"
server_key_version: "encryptionkey@0"
I20260928 16:34:45.205708 1249 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:34:45.208004 1264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:45.209249 1249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:45.209390 1249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "3dd875f68ab248caa944d9d288b91c82"
format_stamp: "Formatted at 2026-09-28 16:34:45 on dist-test-slave-g9sh"
server_key: "efe120510958457e3e32597090339baf"
server_key_iv: "58b967d09512230a00020e6b56fbaf54"
server_key_version: "encryptionkey@0"
I20260928 16:34:45.209496 1249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:45.220378 1249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:45.222959 1249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:45.223150 1249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:45.230019 1249 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:33927
I20260928 16:34:45.230093 1316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:33927 every 8 connection(s)
I20260928 16:34:45.230942 1249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:45.233568 1317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:45.238741 1317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82: Bootstrap starting.
I20260928 16:34:45.241114 1317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:45.241160 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 1249
I20260928 16:34:45.241284 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:45.241566 28139 external_mini_cluster.cc:1428] Setting key c5cb0a7b23726f541418735aba19b185
I20260928 16:34:45.241974 1317 log.cc:824] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:45.244211 1317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82: No bootstrap required, opened a new log
I20260928 16:34:45.248514 1317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd875f68ab248caa944d9d288b91c82" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 33927 } }
I20260928 16:34:45.248780 1317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:45.248848 1317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd875f68ab248caa944d9d288b91c82, State: Initialized, Role: FOLLOWER
I20260928 16:34:45.249341 1317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd875f68ab248caa944d9d288b91c82" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 33927 } }
I20260928 16:34:45.249511 1317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:45.249609 1317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:45.249706 1317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:45.250528 1317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd875f68ab248caa944d9d288b91c82" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 33927 } }
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:45.250906 1317 leader_election.cc:302] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dd875f68ab248caa944d9d288b91c82; no voters:
I20260928 16:34:45.251237 1317 leader_election.cc:288] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:45.251309 1322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:45.251569 1322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [term 1 LEADER]: Becoming Leader. State: Replica: 3dd875f68ab248caa944d9d288b91c82, State: Running, Role: LEADER
I20260928 16:34:45.252285 1322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd875f68ab248caa944d9d288b91c82" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 33927 } }
I20260928 16:34:45.252681 1317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:45.253499 1322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dd875f68ab248caa944d9d288b91c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd875f68ab248caa944d9d288b91c82" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 33927 } } }
I20260928 16:34:45.253691 1322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:45.253937 1323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dd875f68ab248caa944d9d288b91c82. Latest consensus state: current_term: 1 leader_uuid: "3dd875f68ab248caa944d9d288b91c82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dd875f68ab248caa944d9d288b91c82" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 33927 } } }
I20260928 16:34:45.254043 1323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:45.254087 1331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:45.257265 1331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:45.257334 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:45.242660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.27.122.254:33927)
0928 16:34:45.243092 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:45.243111 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:45.243151 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:34:45.243952 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:45.243975 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:45.244362 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:45.244756 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:45.245549 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.247034 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.247840 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.248507 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.248674 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:45.251430 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:45.251437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:45.251461 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:45.251488 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:45.253581 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.254175 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.254181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.254187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.254321 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.254672 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.254675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.254677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.255168 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:45.255943 (+ 775us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:45.256105 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 16:34:45.256401 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
I20260928 16:34:45.267257 1331 catalog_manager.cc:1347] Generated new cluster ID: aaeb7f05e1bd45f19443a527bed72be4
I20260928 16:34:45.267330 1331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:45.283147 1331 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:45.284025 1331 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:45.288205 1331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dd875f68ab248caa944d9d288b91c82: Generated new TSK 0
I20260928 16:34:45.288807 1331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:34:45.323348 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:33927
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:45.433455 1344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:45.433676 1344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:45.433715 1344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:45.437602 1344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:45.437646 1344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:45.437700 1344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:45.442622 1344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:45.443454 1344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:45.444317 1344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:45.453603 1352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:45.453683 1350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:45.453670 1344 server_base.cc:1034] running on GCE node
W20260928 16:34:45.453616 1349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:45.454316 1344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:45.454927 1344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:45.456086 1344 hybrid_clock.cc:648] HybridClock initialized: now 1790613285456054 us; error 44 us; skew 500 ppm
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:45.458793 1344 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:34:45.459580 1344 webserver.cc:460] Webserver started at http://127.27.122.193:43275/ using document root <none> and password file <none>
I20260928 16:34:45.460080 1344 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:45.460139 1344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:45.460295 1344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:45.462344 1344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "bf48b7f4a8fc42b8878f989a7f7e4730"
format_stamp: "Formatted at 2026-09-28 16:34:45 on dist-test-slave-g9sh"
server_key: "b1f61389cf718214e0c1defc213c3f57"
server_key_iv: "4ee5023dd0444bcf9872879385ace5b8"
server_key_version: "encryptionkey@0"
I20260928 16:34:45.462859 1344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "bf48b7f4a8fc42b8878f989a7f7e4730"
format_stamp: "Formatted at 2026-09-28 16:34:45 on dist-test-slave-g9sh"
server_key: "b1f61389cf718214e0c1defc213c3f57"
server_key_iv: "4ee5023dd0444bcf9872879385ace5b8"
server_key_version: "encryptionkey@0"
I20260928 16:34:45.466928 1344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:34:45.469286 1359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:45.470484 1344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:45.470633 1344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "bf48b7f4a8fc42b8878f989a7f7e4730"
format_stamp: "Formatted at 2026-09-28 16:34:45 on dist-test-slave-g9sh"
server_key: "b1f61389cf718214e0c1defc213c3f57"
server_key_iv: "4ee5023dd0444bcf9872879385ace5b8"
server_key_version: "encryptionkey@0"
I20260928 16:34:45.470749 1344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:45.485500 1344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:45.489606 1344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:45.489806 1344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:45.490473 1344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:45.491509 1344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:45.491590 1344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:45.491663 1344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:45.491698 1344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:45.500685 1344 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:45043
I20260928 16:34:45.500711 1473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:45043 every 8 connection(s)
I20260928 16:34:45.501629 1344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:45.508932 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 1344
I20260928 16:34:45.509093 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:34:45.509460 28139 external_mini_cluster.cc:1428] Setting key 9bdc39a3e55ba83ecaebf4d60b16157d
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:45.515632 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:45.503332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:35917 (local address 127.27.122.254:33927)
0928 16:34:45.503467 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:45.503470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:45.504217 (+ 747us) server_negotiation.cc:407] Connection header received
0928 16:34:45.505712 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:45.505716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:45.505777 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:45.505879 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:45.507044 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.507595 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.508736 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.509081 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.509213 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:45.511869 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:45.511876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:45.511882 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:45.511916 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:45.513497 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.514121 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.514125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.514127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.514217 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.514555 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.514559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.514561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.514746 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:45.514852 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:45.515339 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 16:34:45.515463 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:34:45.516414 1476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:45.503639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:33927 (local address 127.27.122.193:35917)
0928 16:34:45.504068 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:45.504114 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:45.505473 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:45.506057 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:45.506064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:45.506420 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:45.506892 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:45.506904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.507726 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.507735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:45.508601 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:45.508610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.509329 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.509334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:45.509550 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:45.510223 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:45.510246 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:45.511708 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:45.513634 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:45.513641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:45.513657 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:45.514012 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:45.514311 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:45.514315 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:45.514317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:45.514454 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:45.514865 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:45.514873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:45.515108 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 16:34:45.515296 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 16:34:45.515610 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":99,"threads_started":1}
I20260928 16:34:45.517622 1474 heartbeater.cc:347] Connected to a master server at 127.27.122.254:33927
I20260928 16:34:45.517892 1474 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:45.518496 1474 heartbeater.cc:511] Master 127.27.122.254:33927 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:34:45.520144 1281 ts_manager.cc:194] Registered new tserver with Master: bf48b7f4a8fc42b8878f989a7f7e4730 (127.27.122.193:45043)
I20260928 16:34:45.521411 1281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:35917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:34:45.535632 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:33927
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:45.644102 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:45.644318 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:45.644393 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:45.648360 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:45.648412 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:45.648466 1479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:45.653885 1479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:45.654883 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:45.655838 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:45.665210 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:45.665246 1479 server_base.cc:1034] running on GCE node
W20260928 16:34:45.665395 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:45.665226 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:45.665928 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:45.666492 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:45.667651 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790613285667631 us; error 31 us; skew 500 ppm
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:45.670423 1479 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:34:45.671274 1479 webserver.cc:460] Webserver started at http://127.27.122.194:46451/ using document root <none> and password file <none>
I20260928 16:34:45.671805 1479 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:45.671895 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:45.672128 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:45.674186 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "7aeb72973e2a4323b09c1945fcc69776"
format_stamp: "Formatted at 2026-09-28 16:34:45 on dist-test-slave-g9sh"
server_key: "3cc5e19abff2c2827c33cbd58f4f6930"
server_key_iv: "a57fb7a57a5464dd74d50aaff9c44d32"
server_key_version: "encryptionkey@0"
I20260928 16:34:45.674737 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "7aeb72973e2a4323b09c1945fcc69776"
format_stamp: "Formatted at 2026-09-28 16:34:45 on dist-test-slave-g9sh"
server_key: "3cc5e19abff2c2827c33cbd58f4f6930"
server_key_iv: "a57fb7a57a5464dd74d50aaff9c44d32"
server_key_version: "encryptionkey@0"
I20260928 16:34:45.678258 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:34:45.680501 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:45.681699 1479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:45.681814 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "7aeb72973e2a4323b09c1945fcc69776"
format_stamp: "Formatted at 2026-09-28 16:34:45 on dist-test-slave-g9sh"
server_key: "3cc5e19abff2c2827c33cbd58f4f6930"
server_key_iv: "a57fb7a57a5464dd74d50aaff9c44d32"
server_key_version: "encryptionkey@0"
I20260928 16:34:45.681910 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:45.693863 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:45.697016 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:45.697206 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:45.697739 1479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:45.698499 1479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:45.698544 1479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:45.698616 1479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:45.698653 1479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:45.706903 1479 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:42519
I20260928 16:34:45.706934 1608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:42519 every 8 connection(s)
I20260928 16:34:45.707737 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:45.711076 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 1479
I20260928 16:34:45.711207 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:34:45.711478 28139 external_mini_cluster.cc:1428] Setting key 16efcbb095d8e8a85619e1ffa565431a
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
I20260928 16:34:45.722805 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:45.709484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:51667 (local address 127.27.122.254:33927)
0928 16:34:45.709621 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:45.709625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:45.710403 (+ 778us) server_negotiation.cc:407] Connection header received
0928 16:34:45.711936 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:45.711939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:45.711987 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:45.712104 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:45.713507 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.714405 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.716105 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.716317 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.716397 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:45.719097 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:45.719119 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:45.719124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:45.719160 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:45.720811 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.721338 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.721344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.721346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.721441 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.721786 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.721789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.721791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.722004 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:45.722110 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:45.722588 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 16:34:45.722690 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:34:45.723556 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:45.709847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:33927 (local address 127.27.122.194:51667)
0928 16:34:45.710267 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:45.710307 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:45.711713 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:45.712288 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:45.712298 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:45.712799 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:45.713364 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:45.713377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.714525 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.714528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:45.715992 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:45.716003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.716464 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.716471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:45.716708 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:45.717296 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:45.717322 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:45.718932 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:45.720972 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:45.720977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:45.720987 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:45.721229 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:45.721549 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:45.721551 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:45.721552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:45.721680 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:45.722104 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:45.722109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:45.722356 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 16:34:45.722498 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 16:34:45.722733 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
I20260928 16:34:45.724829 1609 heartbeater.cc:347] Connected to a master server at 127.27.122.254:33927
I20260928 16:34:45.725104 1609 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:45.725668 1609 heartbeater.cc:511] Master 127.27.122.254:33927 requested a full tablet report, sending...
I20260928 16:34:45.726827 1281 ts_manager.cc:194] Registered new tserver with Master: 7aeb72973e2a4323b09c1945fcc69776 (127.27.122.194:42519)
I20260928 16:34:45.727648 1281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:51667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:34:45.740248 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:33927
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:45.848428 1614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:45.848640 1614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:45.848685 1614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:45.852615 1614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:45.852663 1614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:45.852764 1614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:45.857905 1614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:33927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:45.858795 1614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:45.859724 1614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:34:45.868676 1614 server_base.cc:1034] running on GCE node
W20260928 16:34:45.868639 1620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:45.868711 1622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:45.868619 1619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:45.869326 1614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:45.869843 1614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:45.871029 1614 hybrid_clock.cc:648] HybridClock initialized: now 1790613285871013 us; error 24 us; skew 500 ppm
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:45.873799 1614 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:34:45.874686 1614 webserver.cc:460] Webserver started at http://127.27.122.195:39713/ using document root <none> and password file <none>
I20260928 16:34:45.875264 1614 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:45.875353 1614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:45.875577 1614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:45.877511 1614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "dee341815c6b415ab518e6b95d07a064"
format_stamp: "Formatted at 2026-09-28 16:34:45 on dist-test-slave-g9sh"
server_key: "743d53388f5486753084fcac41df4cbe"
server_key_iv: "0d5046b8c100c0f0290841c12ef64b76"
server_key_version: "encryptionkey@0"
I20260928 16:34:45.878050 1614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "dee341815c6b415ab518e6b95d07a064"
format_stamp: "Formatted at 2026-09-28 16:34:45 on dist-test-slave-g9sh"
server_key: "743d53388f5486753084fcac41df4cbe"
server_key_iv: "0d5046b8c100c0f0290841c12ef64b76"
server_key_version: "encryptionkey@0"
I20260928 16:34:45.881464 1614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:34:45.883705 1629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:45.884847 1614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:45.885003 1614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "dee341815c6b415ab518e6b95d07a064"
format_stamp: "Formatted at 2026-09-28 16:34:45 on dist-test-slave-g9sh"
server_key: "743d53388f5486753084fcac41df4cbe"
server_key_iv: "0d5046b8c100c0f0290841c12ef64b76"
server_key_version: "encryptionkey@0"
I20260928 16:34:45.885121 1614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:45.899904 1614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:45.903021 1614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:45.903230 1614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:45.903781 1614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:45.904538 1614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:45.904605 1614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:45.904664 1614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:45.904716 1614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:45.913048 1614 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:42703
I20260928 16:34:45.913077 1743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:42703 every 8 connection(s)
I20260928 16:34:45.913957 1614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:34:45.915797 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 1614
I20260928 16:34:45.915911 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:34:45.916183 28139 external_mini_cluster.cc:1428] Setting key 5e177912a57eac5f1aaed6866bf56694
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:45.926606 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:45.915685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:51583 (local address 127.27.122.254:33927)
0928 16:34:45.915855 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:45.915860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:45.916655 (+ 795us) server_negotiation.cc:407] Connection header received
0928 16:34:45.918093 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:45.918097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:45.918153 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:45.918227 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:45.919315 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.919851 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.921015 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.921249 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.921363 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:45.923642 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:45.923645 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:45.923647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:45.923671 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:45.924867 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.925376 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.925378 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.925380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.925452 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.925745 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.925747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.925748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.925907 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:45.925982 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:45.926400 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 16:34:45.926488 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 16:34:45.927363 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:45.916071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:33927 (local address 127.27.122.195:51583)
0928 16:34:45.916511 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:45.916564 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:45.917882 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:45.918385 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:45.918392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:45.918711 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:45.919167 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:45.919181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.919977 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.919981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:45.920829 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:45.920839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.921394 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.921398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:45.921563 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:45.922109 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:45.922125 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:45.923494 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:45.925023 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:45.925027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:45.925038 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:45.925290 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:45.925533 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:45.925535 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:45.925536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:45.925666 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:45.925994 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:45.925999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:45.926206 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 16:34:45.926351 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 16:34:45.926574 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":131,"threads_started":1}
I20260928 16:34:45.928370 1744 heartbeater.cc:347] Connected to a master server at 127.27.122.254:33927
I20260928 16:34:45.928584 1744 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:45.929162 1744 heartbeater.cc:511] Master 127.27.122.254:33927 requested a full tablet report, sending...
I20260928 16:34:45.930160 1281 ts_manager.cc:194] Registered new tserver with Master: dee341815c6b415ab518e6b95d07a064 (127.27.122.195:42703)
I20260928 16:34:45.930806 1281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:51583
I20260928 16:34:45.941448 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/test-admin.keytab.
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:45.957144 28139 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 16:34:45.959758 1751 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:34:45.958433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.27.122.193:45043)
0928 16:34:45.958731 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:45.958740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:45.958763 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:34:45.958810 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:45.958815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:45.958987 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:45.959117 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:45.959582 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58398: BlockingRecv error: recv got EOF from 127.0.0.1:58398 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":108,"threads_started":1}
W20260928 16:34:45.959800 28139 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/krb5cc' of type FILE
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:45.961114 28139 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:45.962004 28139 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.193@KRBTEST.COM
I20260928 16:34:45.971127 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:45.962829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.27.122.193:45043)
0928 16:34:45.962954 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:45.962959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:45.963056 (+ 97us) server_negotiation.cc:407] Connection header received
0928 16:34:45.963156 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:45.963158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:45.963196 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:45.963261 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:45.963728 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.964510 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.965386 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.965780 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.965870 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:45.967917 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:45.967925 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:45.967936 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:45.967958 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:45.969530 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.969923 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.969928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.969929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.970010 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.970244 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.970246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.970247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.970471 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:45.970625 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:45.970798 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 16:34:45.970980 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 16:34:45.974735 1751 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:34:45.973853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58418 (local address 127.27.122.193:45043)
0928 16:34:45.973960 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:45.973963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:45.974019 (+ 56us) server_negotiation.cc:407] Connection header received
0928 16:34:45.974149 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:45.974151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:45.974183 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:45.974247 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:45.974618 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58418: BlockingRecv error: recv got EOF from 127.0.0.1:58418 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 16:34:45.974895 28139 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/krb5cc' of type FILE
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:45.975875 28139 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:45.976770 28139 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.193@KRBTEST.COM
I20260928 16:34:45.984556 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:45.977536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58422 (local address 127.27.122.193:45043)
0928 16:34:45.977658 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:45.977665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:45.977774 (+ 109us) server_negotiation.cc:407] Connection header received
0928 16:34:45.977844 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:45.977846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:45.977879 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:45.977940 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:45.978368 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.978846 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.979551 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.979766 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.979839 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:45.981836 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:45.981838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:45.981840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:45.981861 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:45.983219 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.983557 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.983559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.983560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.983638 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.983856 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.983857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.983859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.983993 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:45.984080 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:45.984291 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 16:34:45.984409 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 16:34:45.986935 1751 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:34:45.986138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.27.122.193:45043)
0928 16:34:45.986268 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:45.986272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:45.986320 (+ 48us) server_negotiation.cc:407] Connection header received
0928 16:34:45.986412 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:45.986414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:45.986444 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:45.986505 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:45.986827 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58438: BlockingRecv error: recv got EOF from 127.0.0.1:58438 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 16:34:45.986990 28139 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790613255639823-28139-0/krb5kdc/krb5cc' of type FILE
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:45.987861 28139 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:45.988754 28139 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 16:34:45 dist-test-slave-g9sh krb5kdc[1235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.193@KRBTEST.COM
I20260928 16:34:45.996300 1751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:45.989588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58448 (local address 127.27.122.193:45043)
0928 16:34:45.989687 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:45.989691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:45.989827 (+ 136us) server_negotiation.cc:407] Connection header received
0928 16:34:45.989908 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:45.989910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:45.989939 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:45.989997 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:45.990406 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.990863 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.991562 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:45.991762 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:45.991832 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:45.993927 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:45.993930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:45.993931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:45.993952 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:45.995072 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.995423 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.995426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.995427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.995493 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:45.995717 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:45.995719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:45.995720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:45.995854 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:45.995940 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:45.996129 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 16:34:45.996208 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:34:45.997234 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 1344
I20260928 16:34:46.004261 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 1479
I20260928 16:34:46.009852 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 1614
I20260928 16:34:46.015363 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 1249
2026-09-28T16:34:46Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2108 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:46 dist-test-slave-g9sh krb5kdc[1764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:46 dist-test-slave-g9sh krb5kdc[1764](info): set up 2 sockets
Sep 28 16:34:46 dist-test-slave-g9sh krb5kdc[1764](info): commencing operation
krb5kdc: starting...
W20260928 16:34:47.052318 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:47 dist-test-slave-g9sh krb5kdc[1764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:47Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.254.keytab.
Entry for principal oryx/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.254.keytab.
I20260928 16:34:47.170295 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:41397
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.254.keytab
--principal=oryx/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:41397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:47.276414 1777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:47.276708 1777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:47.276782 1777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:47.280753 1777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:47.280843 1777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:47.280879 1777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:47.280934 1777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:47.280990 1777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:47.286093 1777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:41397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.254.keytab
--principal=oryx/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:41397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:47.287107 1777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:47.288129 1777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:47.296396 1782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:47.296479 1777 server_base.cc:1034] running on GCE node
W20260928 16:34:47.296446 1783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:47.296396 1786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:47.297286 1777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:47.298357 1777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:47.299549 1777 hybrid_clock.cc:648] HybridClock initialized: now 1790613287299538 us; error 26 us; skew 500 ppm
Sep 28 16:34:47 dist-test-slave-g9sh krb5kdc[1764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613287, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:47.302393 1777 init.cc:377] Logged in from keytab as oryx/127.27.122.254@KRBTEST.COM (short username oryx)
I20260928 16:34:47.303244 1777 webserver.cc:460] Webserver started at http://127.27.122.254:40137/ using document root <none> and password file <none>
I20260928 16:34:47.303754 1777 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:47.303812 1777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:47.303987 1777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:47.306125 1777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "7a8e7bdef6c440389e1041f44c744b40"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "e7320eafc4de9667b2041877325ed8ad"
server_key_iv: "7f1d0845c7f44c48d8628940b1c6cd83"
server_key_version: "encryptionkey@0"
I20260928 16:34:47.306667 1777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "7a8e7bdef6c440389e1041f44c744b40"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "e7320eafc4de9667b2041877325ed8ad"
server_key_iv: "7f1d0845c7f44c48d8628940b1c6cd83"
server_key_version: "encryptionkey@0"
I20260928 16:34:47.310441 1777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:34:47.312872 1793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:47.314211 1777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:47.314338 1777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "7a8e7bdef6c440389e1041f44c744b40"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "e7320eafc4de9667b2041877325ed8ad"
server_key_iv: "7f1d0845c7f44c48d8628940b1c6cd83"
server_key_version: "encryptionkey@0"
I20260928 16:34:47.314446 1777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:47.325804 1777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:47.329981 1777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:47.330186 1777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:47.337102 1777 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:41397
I20260928 16:34:47.337404 1845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:41397 every 8 connection(s)
I20260928 16:34:47.338059 1777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:47.340754 1846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:47.345527 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 1777
I20260928 16:34:47.345679 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:47.345944 28139 external_mini_cluster.cc:1428] Setting key cd182485eef4bc4d982e325d1874f287
I20260928 16:34:47.346347 1846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40: Bootstrap starting.
I20260928 16:34:47.348654 1846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:47.349648 1846 log.cc:824] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:47.351658 1846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40: No bootstrap required, opened a new log
Sep 28 16:34:47 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
I20260928 16:34:47.356003 1846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } }
I20260928 16:34:47.356278 1846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:47.356346 1846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a8e7bdef6c440389e1041f44c744b40, State: Initialized, Role: FOLLOWER
I20260928 16:34:47.356843 1846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } }
I20260928 16:34:47.357048 1846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:47.357136 1846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:47.357237 1846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:47.358083 1846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } }
I20260928 16:34:47.358424 1846 leader_election.cc:302] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a8e7bdef6c440389e1041f44c744b40; no voters:
I20260928 16:34:47.358778 1846 leader_election.cc:288] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:47.358817 1851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:47.359067 1851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [term 1 LEADER]: Becoming Leader. State: Replica: 7a8e7bdef6c440389e1041f44c744b40, State: Running, Role: LEADER
I20260928 16:34:47.359577 1851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } }
I20260928 16:34:47.360107 1846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:47.360339 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:47.347297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.27.122.254:41397)
0928 16:34:47.347699 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:47.347725 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:47.347760 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:34:47.348380 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:47.348396 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:47.348713 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:47.349103 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:47.349977 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:47.350951 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:47.352095 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:47.352540 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:47.352658 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:47.355333 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:47.355337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:47.355349 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:47.355375 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:47.357263 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:47.357716 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:47.357722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:47.357728 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:47.357849 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:47.358131 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:47.358135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:47.358137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:47.358620 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:47.358826 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:47.359117 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 16:34:47.359398 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":144,"threads_started":1}
I20260928 16:34:47.362030 1852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a8e7bdef6c440389e1041f44c744b40. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } }
I20260928 16:34:47.362330 1853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } }
I20260928 16:34:47.362429 1852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:47.363273 1853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: This master's current role is: LEADER
W20260928 16:34:47.364976 1865 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:34:47.365052 1865 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:34:47.365134 1868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:47.365975 1868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:47.371522 1868 catalog_manager.cc:1347] Generated new cluster ID: 90365853bc0a4c12a1cac893085786d0
I20260928 16:34:47.371593 1868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:47.382869 1868 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:47.384029 1868 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:47.394701 1868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40: Generated new TSK 0
I20260928 16:34:47.395381 1868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab.
Entry for principal oryx/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab.
I20260928 16:34:47.427456 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41397
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab
--principal=oryx/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:47.537050 1872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:47.537312 1872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:47.537384 1872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:47.541525 1872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:47.541603 1872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:47.541708 1872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:47.546810 1872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab
--principal=oryx/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:47.547801 1872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:47.548802 1872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:47.558204 1878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:47.558226 1872 server_base.cc:1034] running on GCE node
W20260928 16:34:47.558238 1877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:47.558156 1880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:47.558770 1872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:47.559388 1872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:47.560585 1872 hybrid_clock.cc:648] HybridClock initialized: now 1790613287560557 us; error 25 us; skew 500 ppm
Sep 28 16:34:47 dist-test-slave-g9sh krb5kdc[1764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613287, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:47.563380 1872 init.cc:377] Logged in from keytab as oryx/127.27.122.193@KRBTEST.COM (short username oryx)
I20260928 16:34:47.564193 1872 webserver.cc:460] Webserver started at http://127.27.122.193:41065/ using document root <none> and password file <none>
I20260928 16:34:47.564766 1872 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:47.564853 1872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:47.565102 1872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:47.567090 1872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "4e4a75965fcd4f65af7ed425d7d56296"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "15fe17ae5fd34be4c6b833e912a06a52"
server_key_iv: "0bc15d6fe6fd9525a2cb6df5ee0aa9f5"
server_key_version: "encryptionkey@0"
I20260928 16:34:47.567646 1872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "4e4a75965fcd4f65af7ed425d7d56296"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "15fe17ae5fd34be4c6b833e912a06a52"
server_key_iv: "0bc15d6fe6fd9525a2cb6df5ee0aa9f5"
server_key_version: "encryptionkey@0"
I20260928 16:34:47.571223 1872 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 16:34:47.573588 1887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:47.574779 1872 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:34:47.574911 1872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "4e4a75965fcd4f65af7ed425d7d56296"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "15fe17ae5fd34be4c6b833e912a06a52"
server_key_iv: "0bc15d6fe6fd9525a2cb6df5ee0aa9f5"
server_key_version: "encryptionkey@0"
I20260928 16:34:47.575019 1872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:47.597114 1872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:47.599915 1872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:47.600114 1872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:47.600696 1872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:47.601564 1872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:47.601642 1872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:47.601716 1872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:47.601770 1872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:47.610769 1872 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:45537
I20260928 16:34:47.610803 2003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:45537 every 8 connection(s)
I20260928 16:34:47.611637 1872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:47.613833 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 1872
I20260928 16:34:47.613953 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:34:47.614305 28139 external_mini_cluster.cc:1428] Setting key 3fd43d8475f961ceec9219c3388a4078
Sep 28 16:34:47 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613287, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.122.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.122.194@KRBTEST.COM" created.
I20260928 16:34:47.627345 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:47.613397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:48665 (local address 127.27.122.254:41397)
0928 16:34:47.613542 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:47.613547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:47.614572 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 16:34:47.616148 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:47.616152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:47.616206 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:47.616288 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:47.617755 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:47.618351 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:47.619615 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:47.620051 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:47.620183 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:47.623493 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:47.623497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:47.623500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:47.623529 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:47.625142 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:47.625679 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:47.625684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:47.625686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:47.625776 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:47.626184 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:47.626188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:47.626189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:47.626385 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:47.626503 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:47.627043 (+ 540us) server_negotiation.cc:299] Negotiation successful
0928 16:34:47.627166 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:34:47.628279 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:47.613895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.193:48665)
0928 16:34:47.614434 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:47.614478 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:47.615941 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:47.616457 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:47.616464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:47.616812 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:47.617536 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:47.617558 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:47.618480 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:47.618483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:47.619450 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:47.619460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:47.620218 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:47.620223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:47.620484 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:47.621248 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:47.621298 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:47.623325 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:47.625272 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:47.625278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:47.625293 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:47.625570 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:47.625898 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:47.625901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:47.625903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:47.626074 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:47.626502 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:47.626509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:47.626798 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 16:34:47.626987 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 16:34:47.627320 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":153,"threads_started":1}
I20260928 16:34:47.630285 2004 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41397
I20260928 16:34:47.630559 2004 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:47.631065 2004 heartbeater.cc:511] Master 127.27.122.254:41397 requested a full tablet report, sending...
I20260928 16:34:47.632849 1810 ts_manager.cc:194] Registered new tserver with Master: 4e4a75965fcd4f65af7ed425d7d56296 (127.27.122.193:45537)
I20260928 16:34:47.634536 1810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:48665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab.
Entry for principal oryx/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab.
I20260928 16:34:47.643612 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41397
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab
--principal=oryx/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:47.751749 2009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:47.751978 2009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:47.752018 2009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:47.755934 2009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:47.755983 2009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:47.756037 2009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:47.761111 2009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab
--principal=oryx/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:47.761973 2009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:47.762914 2009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:47.772284 2015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:47.772298 2016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:47.772383 2009 server_base.cc:1034] running on GCE node
W20260928 16:34:47.772341 2018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:47.772933 2009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:47.773483 2009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:47.774638 2009 hybrid_clock.cc:648] HybridClock initialized: now 1790613287774622 us; error 28 us; skew 500 ppm
Sep 28 16:34:47 dist-test-slave-g9sh krb5kdc[1764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613287, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:47.777207 2009 init.cc:377] Logged in from keytab as oryx/127.27.122.194@KRBTEST.COM (short username oryx)
I20260928 16:34:47.778110 2009 webserver.cc:460] Webserver started at http://127.27.122.194:43791/ using document root <none> and password file <none>
I20260928 16:34:47.778640 2009 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:47.778700 2009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:47.778991 2009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:47.780998 2009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "d4419df5c25243b2999c0bec329e149c"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "a4ac8a45fb147f87b164ee54f6281080"
server_key_iv: "37c1c9c1234e06328a7cea8589ce6ab3"
server_key_version: "encryptionkey@0"
I20260928 16:34:47.781546 2009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "d4419df5c25243b2999c0bec329e149c"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "a4ac8a45fb147f87b164ee54f6281080"
server_key_iv: "37c1c9c1234e06328a7cea8589ce6ab3"
server_key_version: "encryptionkey@0"
I20260928 16:34:47.785146 2009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260928 16:34:47.787403 2025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:47.788607 2009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:47.788741 2009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "d4419df5c25243b2999c0bec329e149c"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "a4ac8a45fb147f87b164ee54f6281080"
server_key_iv: "37c1c9c1234e06328a7cea8589ce6ab3"
server_key_version: "encryptionkey@0"
I20260928 16:34:47.788847 2009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:47.804425 2009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:47.807126 2009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:47.807324 2009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:47.807870 2009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:47.808655 2009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:47.808724 2009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:47.808795 2009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:47.808838 2009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:47.817610 2009 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:38565
I20260928 16:34:47.817689 2139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:38565 every 8 connection(s)
I20260928 16:34:47.818572 2009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:47.819110 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 2009
I20260928 16:34:47.819239 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:34:47.819554 28139 external_mini_cluster.cc:1428] Setting key 8e86a06fd13e55ad9b4ec47edc023aaa
Sep 28 16:34:47 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613287, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.122.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.122.195@KRBTEST.COM" created.
I20260928 16:34:47.832412 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:47.820385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:52517 (local address 127.27.122.254:41397)
0928 16:34:47.820529 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:47.820534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:47.821548 (+ 1014us) server_negotiation.cc:407] Connection header received
0928 16:34:47.822918 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:47.822922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:47.822977 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:47.823081 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:47.824202 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:47.824730 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:47.825937 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:47.826181 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:47.826297 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:47.828759 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:47.828762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:47.828764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:47.828789 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:47.830151 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:47.830653 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:47.830656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:47.830657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:47.830767 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:47.831184 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:47.831188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:47.831189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:47.831437 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:47.831548 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:47.832008 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 16:34:47.832164 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:34:47.833190 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:47.820854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.194:52517)
0928 16:34:47.821414 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:47.821456 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:47.822719 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:47.823251 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:47.823258 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:47.823604 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:47.824057 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:47.824070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:47.824866 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:47.824870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:47.825806 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:47.825816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:47.826335 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:47.826339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:47.826506 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:47.827148 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:47.827164 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:47.828601 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:47.830304 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:47.830309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:47.830320 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:47.830564 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:47.830870 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:47.830876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:47.830881 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:47.831072 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:47.831547 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:47.831551 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:47.831773 (+ 222us) client_negotiation.cc:769] Sending connection context
0928 16:34:47.831925 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 16:34:47.832205 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":167,"threads_started":1}
I20260928 16:34:47.834471 2140 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41397
I20260928 16:34:47.834806 2140 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:47.835299 2140 heartbeater.cc:511] Master 127.27.122.254:41397 requested a full tablet report, sending...
I20260928 16:34:47.836401 1810 ts_manager.cc:194] Registered new tserver with Master: d4419df5c25243b2999c0bec329e149c (127.27.122.194:38565)
I20260928 16:34:47.837038 1810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:52517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab.
Entry for principal oryx/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab.
I20260928 16:34:47.847036 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41397
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab
--principal=oryx/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:47.955389 2145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:47.955610 2145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:47.955649 2145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:47.959489 2145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:47.959535 2145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:47.959589 2145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:47.964627 2145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab
--principal=oryx/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:47.965514 2145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:47.966450 2145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:47.975521 2153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:47.975559 2151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:47.975576 2150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:47.975728 2145 server_base.cc:1034] running on GCE node
I20260928 16:34:47.976208 2145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:47.976822 2145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:47.978003 2145 hybrid_clock.cc:648] HybridClock initialized: now 1790613287977986 us; error 26 us; skew 500 ppm
Sep 28 16:34:47 dist-test-slave-g9sh krb5kdc[1764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613287, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:47.980846 2145 init.cc:377] Logged in from keytab as oryx/127.27.122.195@KRBTEST.COM (short username oryx)
I20260928 16:34:47.981710 2145 webserver.cc:460] Webserver started at http://127.27.122.195:43523/ using document root <none> and password file <none>
I20260928 16:34:47.982262 2145 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:47.982326 2145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:47.982563 2145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:47.984567 2145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "5910840c736d43018174009dfc1218f6"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "fcbf0aa0762d394dfa75d6e1e53b763f"
server_key_iv: "d023071dfcb4e080dc9284ee9cf0cb43"
server_key_version: "encryptionkey@0"
I20260928 16:34:47.985191 2145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "5910840c736d43018174009dfc1218f6"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "fcbf0aa0762d394dfa75d6e1e53b763f"
server_key_iv: "d023071dfcb4e080dc9284ee9cf0cb43"
server_key_version: "encryptionkey@0"
I20260928 16:34:47.988837 2145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 16:34:47.991237 2160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:47.992399 2145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:34:47.992537 2145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "5910840c736d43018174009dfc1218f6"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "fcbf0aa0762d394dfa75d6e1e53b763f"
server_key_iv: "d023071dfcb4e080dc9284ee9cf0cb43"
server_key_version: "encryptionkey@0"
I20260928 16:34:47.992658 2145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:48.012538 2145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:48.015913 2145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:48.016117 2145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:48.016712 2145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:48.017676 2145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:48.017757 2145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:48.017829 2145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:48.017870 2145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:48.026439 2145 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:44807
I20260928 16:34:48.026558 2274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:44807 every 8 connection(s)
I20260928 16:34:48.027547 2145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:34:48.032712 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 2145
I20260928 16:34:48.032809 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:34:48.033126 28139 external_mini_cluster.cc:1428] Setting key d695208a5c071367d05ffccbcf115c15
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613287, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
I20260928 16:34:48.041188 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.029340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:58465 (local address 127.27.122.254:41397)
0928 16:34:48.029486 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.029490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.030311 (+ 821us) server_negotiation.cc:407] Connection header received
0928 16:34:48.031660 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.031663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.031720 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.031801 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.033089 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.033759 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.035146 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.035441 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.035560 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.037945 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.037948 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.037950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.037977 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.039468 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.039897 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.039899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.039901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.039972 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.040258 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.040260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.040262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.040425 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.040499 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.040971 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.041070 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:34:48.041970 2277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.029730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.195:58465)
0928 16:34:48.030161 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.030202 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.031440 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.031968 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.031974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.032313 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.032895 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.032954 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.033887 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.033890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.035010 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.035022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.035604 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.035608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.035792 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.036294 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.036310 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.037796 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.039564 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.039568 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.039581 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.039814 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.040051 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.040053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.040055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.040181 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.040510 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.040515 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.040730 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.040879 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.041185 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":117,"threads_started":1}
I20260928 16:34:48.043071 2275 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41397
I20260928 16:34:48.043305 2275 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:48.043773 2275 heartbeater.cc:511] Master 127.27.122.254:41397 requested a full tablet report, sending...
I20260928 16:34:48.044744 1810 ts_manager.cc:194] Registered new tserver with Master: 5910840c736d43018174009dfc1218f6 (127.27.122.195:44807)
I20260928 16:34:48.045320 1810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:58465
I20260928 16:34:48.046959 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.122.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.122.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.122.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.253.keytab.
Entry for principal oryx/127.27.122.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.253.keytab.
I20260928 16:34:48.071609 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.253:44047
--webserver_interface=127.27.122.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.253.keytab
--principal=oryx/127.27.122.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:41397,127.27.122.253:44047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:48.182152 2280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:48.182432 2280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:48.182512 2280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:48.186502 2280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:48.186578 2280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:48.186616 2280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:48.186676 2280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:48.186714 2280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:48.192063 2280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:41397,127.27.122.253:44047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.253.keytab
--principal=oryx/127.27.122.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.253:44047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.122.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
W20260928 16:34:48.192304 2280 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.122.254:41397,127.27.122.253:44047'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 16:34:48.193269 2280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:48.194293 2280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:48.202754 2285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:48.202783 2288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:48.202824 2286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:48.203135 2280 server_base.cc:1034] running on GCE node
I20260928 16:34:48.203696 2280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:48.204789 2280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:48.205979 2280 hybrid_clock.cc:648] HybridClock initialized: now 1790613288205961 us; error 29 us; skew 500 ppm
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613288, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:48.215404 2280 init.cc:377] Logged in from keytab as oryx/127.27.122.253@KRBTEST.COM (short username oryx)
I20260928 16:34:48.216449 2280 webserver.cc:460] Webserver started at http://127.27.122.253:40553/ using document root <none> and password file <none>
I20260928 16:34:48.217103 2280 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:48.217199 2280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:48.217433 2280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:48.219667 2280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/data/instance:
uuid: "1acf860775654f52a463c0180f8c44b8"
format_stamp: "Formatted at 2026-09-28 16:34:48 on dist-test-slave-g9sh"
server_key: "350df38300bf1f7fb84786f0fd09e066"
server_key_iv: "b12770f4bcb93d2eacdbb139051cf788"
server_key_version: "encryptionkey@0"
I20260928 16:34:48.220257 2280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/wal/instance:
uuid: "1acf860775654f52a463c0180f8c44b8"
format_stamp: "Formatted at 2026-09-28 16:34:48 on dist-test-slave-g9sh"
server_key: "350df38300bf1f7fb84786f0fd09e066"
server_key_iv: "b12770f4bcb93d2eacdbb139051cf788"
server_key_version: "encryptionkey@0"
I20260928 16:34:48.223924 2280 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:34:48.226423 2295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:48.227633 2280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260928 16:34:48.227777 2280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/wal
uuid: "1acf860775654f52a463c0180f8c44b8"
format_stamp: "Formatted at 2026-09-28 16:34:48 on dist-test-slave-g9sh"
server_key: "350df38300bf1f7fb84786f0fd09e066"
server_key_iv: "b12770f4bcb93d2eacdbb139051cf788"
server_key_version: "encryptionkey@0"
I20260928 16:34:48.227887 2280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:48.252105 2280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:48.255018 2280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:48.255228 2280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:48.264292 2280 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.253:44047
I20260928 16:34:48.264518 2347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.253:44047 every 8 connection(s)
I20260928 16:34:48.265749 2280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/data/info.pb
I20260928 16:34:48.267226 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 2280
I20260928 16:34:48.269631 2348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:48.274338 2348 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.122.253@KRBTEST.COM
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613288, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.253@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
I20260928 16:34:48.287148 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.269600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.27.122.253:44047)
0928 16:34:48.270183 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.270231 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.270273 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:34:48.271196 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.271218 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.271637 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.271993 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.273197 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.274687 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.275938 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.276639 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.276824 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.280707 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.280714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.280725 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.280759 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.282840 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.283338 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.283344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.283349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.283508 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.283891 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.283894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.283895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.284392 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.285444 (+ 1052us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.285615 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.285988 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":203,"threads_started":1}
I20260928 16:34:48.289701 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.276223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:51666)
0928 16:34:48.277133 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.277204 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.277373 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.277758 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.277762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.278231 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.278362 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.278368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.279329 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.279334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.280260 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.280270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.280996 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.281000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.281145 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.281630 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.281650 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.283638 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.285727 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.285739 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.285748 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.286200 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.288202 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.288208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.288211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.288328 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.288883 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.288892 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.289073 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.289206 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.289400 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":111,"threads_started":1}
I20260928 16:34:48.290319 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.276468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.27.122.254:41397)
0928 16:34:48.276591 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.276602 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.277280 (+ 678us) server_negotiation.cc:407] Connection header received
0928 16:34:48.277496 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.277499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.277558 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.277605 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.278487 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.279154 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.280385 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.280736 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.280847 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.283784 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.283788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.283790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.283818 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.285627 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.288003 (+ 2376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.288006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.288008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.288100 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.288466 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.288470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.288471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.288765 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.288866 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.290012 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.290153 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 16:34:48.291538 2348 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613288, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.253@KRBTEST.COM for oryx/127.27.122.253@KRBTEST.COM
I20260928 16:34:48.301028 2352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.292110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:44047 (local address 127.0.0.1:48850)
0928 16:34:48.292309 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.292324 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.292406 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.292600 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.292603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.292761 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.293415 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.293426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.294127 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.294132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.294647 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.294653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.295108 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.295111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.295235 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.295837 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.295854 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.297459 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.299467 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.299470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.299472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.299718 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.299975 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.299979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.299980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.300098 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.300479 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.300483 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.300568 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.300674 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.300806 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":6}
I20260928 16:34:48.301527 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.292149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.27.122.253:44047)
0928 16:34:48.292310 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.292315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.292367 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:34:48.292473 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.292477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.292510 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.292588 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.293526 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.294015 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.294759 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.294995 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.295112 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.297614 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.297617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.297619 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.297642 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.299369 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.299796 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.299802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.299804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.299879 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.300203 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.300206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.300208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.300371 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.300474 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.300772 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.301333 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":90}
I20260928 16:34:48.304376 2348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: Bootstrap starting.
I20260928 16:34:48.306704 2348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:48.307466 2348 log.cc:824] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:48.309450 2348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: No bootstrap required, opened a new log
I20260928 16:34:48.312381 2348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } }
I20260928 16:34:48.312610 2348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:48.312677 2348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1acf860775654f52a463c0180f8c44b8, State: Initialized, Role: FOLLOWER
I20260928 16:34:48.313197 2348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } }
I20260928 16:34:48.313704 2354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } } }
I20260928 16:34:48.313858 2354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:34:48.314505 2348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:34:48.319243 2365 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:34:48.319319 2365 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:34:48.321305 2280 master_runner.cc:428] Detected that this master 1acf860775654f52a463c0180f8c44b8 is joining an existing cluster
I20260928 16:34:48.321364 2280 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.122.253:44047
I20260928 16:34:48.332571 2373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.323591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:51670)
0928 16:34:48.324066 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.324079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.324171 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.324513 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.324516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.324750 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.324973 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.324985 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.325833 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.325839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.326747 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.326756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.327382 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.327387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.327463 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.328022 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.328035 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.328547 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.330735 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.330738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.330740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.331113 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.331429 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.331432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.331434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.331515 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.331975 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.331980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.332099 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.332233 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.332373 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":27,"thread_start_us":140,"threads_started":1}
I20260928 16:34:48.332988 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.323913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.27.122.253:44047)
0928 16:34:48.324006 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.324010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.324422 (+ 412us) server_negotiation.cc:407] Connection header received
0928 16:34:48.324592 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.324600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.324632 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.324714 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.325288 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.326043 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.327072 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.327298 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.327410 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.329557 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.329561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.329563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.329593 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.331251 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.331658 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.331662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.331664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.331763 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.332016 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.332018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.332020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.332260 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.332352 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.332711 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.332829 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":28}
I20260928 16:34:48.332613 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.323707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.27.122.254:41397)
0928 16:34:48.323840 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.323844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.324169 (+ 325us) server_negotiation.cc:407] Connection header received
0928 16:34:48.324242 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.324245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.324299 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.324385 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.325130 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.325722 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.326875 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.327245 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.327361 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.328723 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.328730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.328733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.328762 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.330607 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.331228 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.331232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.331234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.331324 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.331615 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.331618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.331620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.331835 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.331905 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.332306 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.332441 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:34:48.333461 2372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.323622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:44047 (local address 127.0.0.1:48864)
0928 16:34:48.324352 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.324366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.324473 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.324727 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.324730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.325027 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.325171 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.325178 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.326171 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.326174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.326969 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.326976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.327510 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.327515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.327660 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.328204 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.328217 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.329431 (+ 1214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.331390 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.331392 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.331394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.331562 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.331839 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.331843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.331845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.331920 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.332436 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.332438 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.332523 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.333151 (+ 628us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.333273 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"spinlock_wait_cycles":1920,"thread_start_us":112,"threads_started":1}
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613287, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.254@KRBTEST.COM for oryx/127.27.122.253@KRBTEST.COM
I20260928 16:34:48.348735 2351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.337658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.27.122.253:44047)
0928 16:34:48.337749 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.337753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.338557 (+ 804us) server_negotiation.cc:407] Connection header received
0928 16:34:48.338729 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.338732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.338784 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.338868 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.339613 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.340334 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.342102 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.342444 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.342542 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.345280 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.345284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.345287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.345311 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.346902 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.347441 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.347444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.347446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.347523 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.347810 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.347813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.347815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.348035 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.348131 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.348485 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.348594 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 16:34:48.348767 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.337602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:44047 (local address 127.0.0.1:48868)
0928 16:34:48.338422 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.338469 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.338599 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.338915 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.338921 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.339361 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.339489 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.339497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.340452 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.340455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.341968 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.341983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.342573 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.342580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.342731 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.343257 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.343277 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.345145 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.347022 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.347032 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.347038 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.347338 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.347620 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.347623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.347625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.347716 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.348143 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.348149 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.348272 (+ 123us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.348420 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.348600 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":97,"threads_started":1}
I20260928 16:34:48.350176 1810 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.122.253:44047: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1acf860775654f52a463c0180f8c44b8"
member_type: NON_VOTER
last_known_addr {
host: "127.27.122.253"
port: 44047
}
attrs {
promote: true
}
}
dest_uuid: "7a8e7bdef6c440389e1041f44c744b40"
cas_config_opid_index: -1
I20260928 16:34:48.350984 1810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } }
I20260928 16:34:48.353417 2322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:48.355007 2322 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 FOLLOWER]: Refusing update from remote peer 7a8e7bdef6c440389e1041f44c744b40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 16:34:48.355792 2376 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:48.355810 2378 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1acf860775654f52a463c0180f8c44b8 (127.27.122.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } } }
WARNING: no policy specified for oryx/127.27.122.252@KRBTEST.COM; defaulting to no policy
I20260928 16:34:48.358479 2378 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.122.253:44047
I20260928 16:34:48.358695 2378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } } }
I20260928 16:34:48.358793 2378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:48.358929 2280 master.cc:537] Master@127.27.122.253:44047 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.122.252@KRBTEST.COM" created.
I20260928 16:34:48.362421 2322 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1acf860775654f52a463c0180f8c44b8 (127.27.122.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } } }
I20260928 16:34:48.365463 2354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a8e7bdef6c440389e1041f44c744b40. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } } }
I20260928 16:34:48.365584 2354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:34:48.365737 2383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } } }
I20260928 16:34:48.365815 2383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:34:48.365988 2384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } } }
I20260928 16:34:48.366166 2384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:34:48.367772 2380 mvcc.cc:205] Tried to move back new op lower bound from 7334352025164869632 to 7334352025026043904. Current Snapshot: MvccSnapshot[applied={T|T < 7334352025063497728}]
I20260928 16:34:48.369041 2280 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 LEARNER]: Raft consensus shutting down.
I20260928 16:34:48.369325 2280 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 LEARNER]: Raft consensus is shut down!
I20260928 16:34:48.369410 2280 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.122.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.252.keytab.
Entry for principal oryx/127.27.122.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.252.keytab.
I20260928 16:34:48.385313 2280 master.cc:552] Master@127.27.122.253:44047 shutdown complete.
I20260928 16:34:48.385385 2280 master_runner.cc:305] Clearing existing system tablet
I20260928 16:34:48.385627 2280 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:48.385701 2280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:48.385727 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.252:43451
--webserver_interface=127.27.122.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.252.keytab
--principal=oryx/127.27.122.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:41397,127.27.122.253:44047,127.27.122.252:43451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:34:48.386955 2280 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 16:34:48.387873 2390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:48.388096 2280 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:48.388233 2280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:48.388268 2280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/wal
uuid: "1acf860775654f52a463c0180f8c44b8"
format_stamp: "Formatted at 2026-09-28 16:34:48 on dist-test-slave-g9sh"
server_key: "350df38300bf1f7fb84786f0fd09e066"
server_key_iv: "b12770f4bcb93d2eacdbb139051cf788"
server_key_version: "encryptionkey@0"
I20260928 16:34:48.390153 2280 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:48.392515 2280 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 16:34:48.392603 2280 log.cc:1196] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 16:34:48.392889 2280 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: Deleting consensus metadata
I20260928 16:34:48.393172 2280 master_runner.cc:315] Copying system tablet from 127.27.122.254:41397
I20260928 16:34:48.394390 2280 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: tablet copy: Beginning tablet copy session from remote peer at address 127.27.122.254:41397
I20260928 16:34:48.404060 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.395085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51682 (local address 127.27.122.254:41397)
0928 16:34:48.395212 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.395217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.395436 (+ 219us) server_negotiation.cc:407] Connection header received
0928 16:34:48.395645 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.395648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.395700 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.395754 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.396444 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.397361 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.398599 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.398963 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.399034 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.400467 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.400471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.400473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.400500 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.402245 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.402752 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.402756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.402758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.402845 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.403136 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.403139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.403141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.403389 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.403493 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.403773 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.403890 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:34:48.404346 2396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.394982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:51682)
0928 16:34:48.395342 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.395357 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.395515 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.395952 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.395955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.396174 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.396300 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.396307 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.397540 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.397543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.398468 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.398477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.399207 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.399210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.399306 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.399879 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.399894 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.400314 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.402356 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.402359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.402361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.402642 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.402952 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.402955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.402956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.403047 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.403489 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.403492 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.403596 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.403699 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.403835 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":94,"threads_started":1}
I20260928 16:34:48.405004 1830 tablet_copy_service.cc:140] P 7a8e7bdef6c440389e1041f44c744b40: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1acf860775654f52a463c0180f8c44b8 ({username='oryx', principal='oryx/127.27.122.253@KRBTEST.COM'} at 127.0.0.1:51682)
I20260928 16:34:48.405189 1830 tablet_copy_service.cc:161] P 7a8e7bdef6c440389e1041f44c744b40: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1acf860775654f52a463c0180f8c44b8 at {username='oryx', principal='oryx/127.27.122.253@KRBTEST.COM'} at 127.0.0.1:51682: session id = 1acf860775654f52a463c0180f8c44b8-00000000000000000000000000000000
I20260928 16:34:48.406998 1830 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 16:34:48.408679 2280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 16:34:48.410913 2280 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: tablet copy: Starting download of 0 data blocks...
I20260928 16:34:48.411106 2280 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: tablet copy: Starting download of 1 WAL segments...
I20260928 16:34:48.412796 2280 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 16:34:48.414590 1830 tablet_copy_service.cc:342] P 7a8e7bdef6c440389e1041f44c744b40: Request end of tablet copy session 1acf860775654f52a463c0180f8c44b8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.122.253@KRBTEST.COM'} at 127.0.0.1:51682
I20260928 16:34:48.414701 1830 tablet_copy_service.cc:433] P 7a8e7bdef6c440389e1041f44c744b40: ending tablet copy session 1acf860775654f52a463c0180f8c44b8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1acf860775654f52a463c0180f8c44b8
I20260928 16:34:48.422487 2280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:48.424250 2397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:48.424437 2398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:48.424819 2280 server_base.cc:1034] running on GCE node
W20260928 16:34:48.424986 2400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:48.425217 2280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:48.425509 2280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:48.426658 2280 hybrid_clock.cc:648] HybridClock initialized: now 1790613288426644 us; error 28 us; skew 500 ppm
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613288, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:48.428362 2280 init.cc:377] Logged in from keytab as oryx/127.27.122.253@KRBTEST.COM (short username oryx)
I20260928 16:34:48.428828 2280 webserver.cc:460] Webserver started at http://127.27.122.253:32989/ using document root <none> and password file <none>
I20260928 16:34:48.429068 2280 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:48.429116 2280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:48.430397 2280 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:48.431386 2407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:48.431661 2280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:48.431738 2280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/wal
uuid: "1acf860775654f52a463c0180f8c44b8"
format_stamp: "Formatted at 2026-09-28 16:34:48 on dist-test-slave-g9sh"
server_key: "350df38300bf1f7fb84786f0fd09e066"
server_key_iv: "b12770f4bcb93d2eacdbb139051cf788"
server_key_version: "encryptionkey@0"
I20260928 16:34:48.431816 2280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:48.452255 2280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:48.454540 2280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:48.459586 2280 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.253:44047
I20260928 16:34:48.459697 2459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.253:44047 every 8 connection(s)
I20260928 16:34:48.460163 2280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-1/data/info.pb
I20260928 16:34:48.462230 2460 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:34:48.463335 2460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: Bootstrap starting.
I20260928 16:34:48.471236 2460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:34:48.471555 2460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: Bootstrap complete.
I20260928 16:34:48.472118 2460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } }
I20260928 16:34:48.472273 2460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1acf860775654f52a463c0180f8c44b8, State: Initialized, Role: LEARNER
I20260928 16:34:48.472399 2460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } }
I20260928 16:34:48.472746 2463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } } }
I20260928 16:34:48.472862 2460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:48.472941 2463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:34:48.477155 2474 catalog_manager.cc:1259] Loaded cluster ID: 90365853bc0a4c12a1cac893085786d0
I20260928 16:34:48.477216 2474 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: loading cluster ID for follower catalog manager: success
I20260928 16:34:48.479315 2474 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: acquiring CA information for follower catalog manager: success
I20260928 16:34:48.480702 2474 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 16:34:48.524732 2389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:48.525043 2389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:48.525117 2389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:48.529109 2389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:48.529158 2389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:48.529177 2389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:48.529193 2389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:48.529248 2389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:48.534493 2389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:41397,127.27.122.253:44047,127.27.122.252:43451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.252.keytab
--principal=oryx/127.27.122.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.252:43451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.122.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:48.535506 2389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:48.536644 2389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:48.545266 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:48.545351 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:48.545306 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:48.545393 2389 server_base.cc:1034] running on GCE node
I20260928 16:34:48.545949 2389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:48.546999 2389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:48.548188 2389 hybrid_clock.cc:648] HybridClock initialized: now 1790613288548174 us; error 29 us; skew 500 ppm
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613288, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:48.550944 2389 init.cc:377] Logged in from keytab as oryx/127.27.122.252@KRBTEST.COM (short username oryx)
I20260928 16:34:48.551738 2389 webserver.cc:460] Webserver started at http://127.27.122.252:37327/ using document root <none> and password file <none>
I20260928 16:34:48.552232 2389 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:48.552287 2389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:48.552450 2389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:48.554440 2389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/data/instance:
uuid: "b414e4445ad146ffb2452cf8ef09e8f7"
format_stamp: "Formatted at 2026-09-28 16:34:48 on dist-test-slave-g9sh"
server_key: "fc7829ce584e9bbfd81a63d99a5c66fb"
server_key_iv: "2c3b84342fc30bfcc2128b6c56891bd9"
server_key_version: "encryptionkey@0"
I20260928 16:34:48.554948 2389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/wal/instance:
uuid: "b414e4445ad146ffb2452cf8ef09e8f7"
format_stamp: "Formatted at 2026-09-28 16:34:48 on dist-test-slave-g9sh"
server_key: "fc7829ce584e9bbfd81a63d99a5c66fb"
server_key_iv: "2c3b84342fc30bfcc2128b6c56891bd9"
server_key_version: "encryptionkey@0"
I20260928 16:34:48.558575 2389 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:34:48.561028 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:48.562268 2389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:34:48.562390 2389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/wal
uuid: "b414e4445ad146ffb2452cf8ef09e8f7"
format_stamp: "Formatted at 2026-09-28 16:34:48 on dist-test-slave-g9sh"
server_key: "fc7829ce584e9bbfd81a63d99a5c66fb"
server_key_iv: "2c3b84342fc30bfcc2128b6c56891bd9"
server_key_version: "encryptionkey@0"
I20260928 16:34:48.562489 2389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:48.578984 2389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:48.582147 2389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:48.582337 2389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:48.589679 2389 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.252:43451
I20260928 16:34:48.589959 2542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.252:43451 every 8 connection(s)
I20260928 16:34:48.590719 2389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/data/info.pb
I20260928 16:34:48.593454 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:48.596803 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 2389
I20260928 16:34:48.597421 2543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613288, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.252@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.122.252@KRBTEST.COM
I20260928 16:34:48.610342 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.599360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.27.122.254:41397)
0928 16:34:48.599509 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.599513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.599880 (+ 367us) server_negotiation.cc:407] Connection header received
0928 16:34:48.600079 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.600086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.600138 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.600218 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.601321 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.602125 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.603510 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.603861 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.603965 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.606684 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.606690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.606693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.606721 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.608092 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.608634 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.608638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.608640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.608727 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.609100 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.609106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.609108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.609344 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.609419 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.610058 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.610175 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 16:34:48.611826 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.598289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32776 (local address 127.27.122.252:43451)
0928 16:34:48.598684 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.598701 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.598737 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:34:48.599523 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.599550 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.599906 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.600146 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.601112 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.602366 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.603500 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.604076 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.604163 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.607213 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.607220 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.607238 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.607260 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.609178 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.609630 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.609635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.609636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.609706 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.609996 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.609999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.610000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.610279 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.610469 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.610934 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.611036 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
I20260928 16:34:48.611826 2546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.599190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:51696)
0928 16:34:48.599730 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.599778 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.599918 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.600296 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.600302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.600804 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.601188 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.601201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.602246 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.602249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.603397 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.603410 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.603996 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.603999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.604166 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.604846 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.604889 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.606542 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.608192 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.608205 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.608217 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.608528 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.608820 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.608822 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.608824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.609008 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.609455 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.609464 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.609785 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.609981 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.610235 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"spinlock_wait_cycles":2304,"thread_start_us":121,"threads_started":1}
I20260928 16:34:48.613358 2543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613288, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.252@KRBTEST.COM for oryx/127.27.122.253@KRBTEST.COM
I20260928 16:34:48.621153 2546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.613833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:44047 (local address 127.0.0.1:48884)
0928 16:34:48.613918 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.613933 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.613999 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.614496 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.614499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.614660 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.614737 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.614742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.615613 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.615619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.616149 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.616156 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.616621 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.616625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.616723 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.617131 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.617142 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.618388 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.619947 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.619949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.619951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.620137 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.620369 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.620371 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.620373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.620442 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.620744 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.620746 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.620814 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.620893 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.621051 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 16:34:48.621222 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.614002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.27.122.253:44047)
0928 16:34:48.614285 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.614290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.614310 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:34:48.614346 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.614348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.614385 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.614483 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.614850 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.615500 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.616270 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.616510 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.616624 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.618511 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.618513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.618515 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.618537 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.619848 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.620219 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.620222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.620223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.620293 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.620514 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.620516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.620517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.620667 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.620739 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.621010 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.621111 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":101,"threads_started":1}
I20260928 16:34:48.621922 2543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613288, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.252@KRBTEST.COM for oryx/127.27.122.252@KRBTEST.COM
I20260928 16:34:48.629792 2546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.622325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:43451 (local address 127.0.0.1:32782)
0928 16:34:48.622425 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.622439 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.622511 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.622735 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.622737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.622875 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.622929 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.622934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.623802 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.623806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.624570 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.624581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.625213 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.625217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.625350 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.625756 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.625765 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.627016 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.628378 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.628384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.628386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.628564 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.628839 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.628843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.628844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.628974 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.629316 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.629318 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.629395 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.629503 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.629637 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 16:34:48.629812 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.622371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.27.122.252:43451)
0928 16:34:48.622478 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.622483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.622496 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:34:48.622606 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.622609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.622647 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.622716 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.623036 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.623691 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.624711 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.625086 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.625197 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.627142 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.627145 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.627147 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.627169 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.628273 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.628665 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.628671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.628673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.628749 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.629056 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.629059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.629060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.629220 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.629322 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.629556 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.629690 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":42}
I20260928 16:34:48.633008 2543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: Bootstrap starting.
I20260928 16:34:48.635108 2543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:48.635772 2543 log.cc:824] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:48.636965 2004 heartbeater.cc:503] Master 127.27.122.254:41397 was elected leader, sending a full tablet report...
I20260928 16:34:48.637773 2543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: No bootstrap required, opened a new log
I20260928 16:34:48.640965 2543 raft_consensus.cc:348] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } }
I20260928 16:34:48.641166 2543 raft_consensus.cc:374] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:48.641243 2543 raft_consensus.cc:729] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b414e4445ad146ffb2452cf8ef09e8f7, State: Initialized, Role: FOLLOWER
I20260928 16:34:48.641636 2543 consensus_queue.cc:260] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } }
I20260928 16:34:48.642052 2549 sys_catalog.cc:455] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } } }
I20260928 16:34:48.642194 2549 sys_catalog.cc:458] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:34:48.642755 2543 sys_catalog.cc:564] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:34:48.647063 2560 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:34:48.647147 2560 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:34:48.650339 2389 master_runner.cc:428] Detected that this master b414e4445ad146ffb2452cf8ef09e8f7 is joining an existing cluster
I20260928 16:34:48.650398 2389 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.122.252:43451
I20260928 16:34:48.661929 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.652943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48900 (local address 127.27.122.253:44047)
0928 16:34:48.653049 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.653053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.653359 (+ 306us) server_negotiation.cc:407] Connection header received
0928 16:34:48.653504 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.653507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.653550 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.653608 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.654516 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.655298 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.656803 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.657173 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.657246 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.658695 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.658698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.658701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.658725 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.660219 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.660624 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.660629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.660631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.660739 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.661139 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.661143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.661144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.661317 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.661422 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.661676 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.661793 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 16:34:48.661929 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.652598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:44047 (local address 127.0.0.1:48900)
0928 16:34:48.653287 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.653300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.653403 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.653738 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.653741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.654179 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.654399 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.654405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.655438 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.655442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.656667 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.656678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.657694 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.657697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.657761 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.658219 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.658230 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.658542 (+ 312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.660327 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.660330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.660332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.660521 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.660831 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.660834 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.660836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.661029 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.661411 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.661414 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.661509 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.661626 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.661754 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":203,"thread_start_us":69,"threads_started":1}
I20260928 16:34:48.665463 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 1872
I20260928 16:34:48.673125 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:45537
--local_ip_for_outbound_sockets=127.27.122.193
--tserver_master_addrs=127.27.122.254:41397
--webserver_port=41065
--webserver_interface=127.27.122.193
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab
--principal=oryx/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.122.254:41397,127.27.122.253:44047
--tserver_master_addrs=127.27.122.254:41397,127.27.122.253:44047,127.27.122.252:43451 with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:34:48.689105 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.652694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32784 (local address 127.27.122.252:43451)
0928 16:34:48.652789 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.652793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.653423 (+ 630us) server_negotiation.cc:407] Connection header received
0928 16:34:48.653574 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.653577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.653611 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.653656 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.654514 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.655274 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.656534 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.656849 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.656948 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.685534 (+ 28586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.685538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.685540 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.685577 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.687476 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.687915 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.687918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.687920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.687983 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.688236 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.688239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.688241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.688442 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.688506 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.688794 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.688939 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 16:34:48.689814 2568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.652865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:43451 (local address 127.0.0.1:32784)
0928 16:34:48.653344 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.653359 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.653485 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.653771 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.653776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.654178 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.654394 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.654403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.655397 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.655400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.656411 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.656423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.657103 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.657106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.657194 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.657704 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.657720 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.685371 (+ 27651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.687598 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.687601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.687603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.687825 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.688069 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.688072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.688075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.688152 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.688560 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.688563 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.688650 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.689490 (+ 840us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.689642 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":66,"thread_start_us":122,"threads_started":1}
I20260928 16:34:48.691632 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.652944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51712 (local address 127.27.122.254:41397)
0928 16:34:48.653049 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.653053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.653331 (+ 278us) server_negotiation.cc:407] Connection header received
0928 16:34:48.653504 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.653507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.653550 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.653602 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.654516 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.655305 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.656803 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.657173 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.657246 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.687605 (+ 30359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.687610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.687614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.687649 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.689579 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.690171 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.690178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.690183 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.690288 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.690663 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.690666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.690668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.690894 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.690992 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.691351 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.691471 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:34:48.691653 2566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.652571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:51712)
0928 16:34:48.653248 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.653262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.653399 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.653738 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.653741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.654134 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.654377 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.654386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.655438 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.655442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.656667 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.656678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.657398 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.657402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.657509 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.686736 (+ 29227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.686755 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.687386 (+ 631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.689726 (+ 2340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.689729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.689732 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.690022 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.690443 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.690447 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.690449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.690548 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.691039 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.691042 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.691153 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.691265 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.691457 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":212,"thread_start_us":91,"threads_started":1}
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613287, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.254@KRBTEST.COM for oryx/127.27.122.252@KRBTEST.COM
I20260928 16:34:48.705477 2545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.696491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.27.122.252:43451)
0928 16:34:48.696624 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.696629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.696643 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:34:48.696713 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.696716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.696772 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.696857 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.697392 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.698131 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.699263 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.699595 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.699714 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.702020 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.702024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.702027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.702051 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.703654 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.704178 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.704181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.704183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.704262 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.704590 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.704593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.704595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.704780 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.704871 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.705237 (+ 366us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.705340 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:34:48.706329 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.695990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:43451 (local address 127.0.0.1:32798)
0928 16:34:48.696132 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.696150 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.696236 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.696928 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.696931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.697131 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.697252 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.697259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.698254 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.698258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.699134 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.699145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.699725 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.699729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.699833 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.700291 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.700305 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.701882 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.703800 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.703804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.703806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.704073 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.704370 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.704373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.704375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.704480 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.704889 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.704895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.705047 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.705173 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.706158 (+ 985us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 16:34:48.707336 1810 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.122.252:43451: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7"
member_type: NON_VOTER
last_known_addr {
host: "127.27.122.252"
port: 43451
}
attrs {
promote: true
}
}
dest_uuid: "7a8e7bdef6c440389e1041f44c744b40"
cas_config_opid_index: 5
I20260928 16:34:48.707976 1810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } }
I20260928 16:34:48.709144 2376 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b414e4445ad146ffb2452cf8ef09e8f7 (127.27.122.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } } }
I20260928 16:34:48.710745 2376 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.122.252:43451
I20260928 16:34:48.710948 2376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } } }
I20260928 16:34:48.711086 2376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:48.711707 2389 master.cc:537] Master@127.27.122.252:43451 shutting down...
I20260928 16:34:48.711906 2517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:48.713548 2517 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 FOLLOWER]: Refusing update from remote peer 7a8e7bdef6c440389e1041f44c744b40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 16:34:48.714133 2377 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:34:48.715538 1797 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 16:34:48.716774 2389 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:48.717053 2389 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:48.717147 2389 tablet_replica.cc:330] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: stopping tablet replica
I20260928 16:34:48.718783 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.709487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.27.122.253:44047)
0928 16:34:48.709604 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.709608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.711826 (+ 2218us) server_negotiation.cc:407] Connection header received
0928 16:34:48.712049 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.712053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.712130 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.712211 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:48.712660 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.713448 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.717123 (+ 3675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.717829 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.717960 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.718098 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.718574 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.718656 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:34:48.719488 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.709263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:44047 (local address 127.0.0.1:48912)
0928 16:34:48.711743 (+ 2480us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.711760 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.711901 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.712407 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.712410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.712430 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:48.712519 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.712525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.713618 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.713621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.716952 (+ 3331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.716966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.718286 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.718290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.718397 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.718441 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.719185 (+ 744us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.719310 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2408}
I20260928 16:34:48.720026 2434 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 LEARNER]: Refusing update from remote peer 7a8e7bdef6c440389e1041f44c744b40: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 16:34:48.720510 2377 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 16:34:48.721829 2434 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b414e4445ad146ffb2452cf8ef09e8f7 (127.27.122.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } } }
I20260928 16:34:48.723686 2463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a8e7bdef6c440389e1041f44c744b40. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } } }
I20260928 16:34:48.723820 2463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:34:48.723920 2463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: NON_VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: true } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } } }
I20260928 16:34:48.724036 2463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:34:48.724509 2377 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40: attempting to promote NON_VOTER 1acf860775654f52a463c0180f8c44b8 to VOTER
I20260928 16:34:48.724921 2377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } }
I20260928 16:34:48.725024 2389 master.cc:552] Master@127.27.122.252:43451 shutdown complete.
I20260928 16:34:48.725062 2389 master_runner.cc:305] Clearing existing system tablet
I20260928 16:34:48.725288 2389 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:48.725330 2389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:48.726835 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.725941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:43451 (local address 127.0.0.1:32812)
0928 16:34:48.726685 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.726756 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":88,"threads_started":1}
I20260928 16:34:48.726814 2434 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 LEARNER]: Refusing update from remote peer 7a8e7bdef6c440389e1041f44c744b40: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 16:34:48.727296 2376 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 16:34:48.727843 2389 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260928 16:34:48.728871 2378 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1acf860775654f52a463c0180f8c44b8 (127.27.122.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } } }
I20260928 16:34:48.729375 2434 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1acf860775654f52a463c0180f8c44b8 (127.27.122.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } } }
I20260928 16:34:48.729563 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.729285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:43451 (local address 127.0.0.1:32814)
0928 16:34:48.729434 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.729488 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 16:34:48.730288 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.730009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:43451 (local address 127.0.0.1:32820)
0928 16:34:48.730147 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.730207 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:43451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260928 16:34:48.730406 1797 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 -> Peer b414e4445ad146ffb2452cf8ef09e8f7 (127.27.122.252:43451): Couldn't send request to peer b414e4445ad146ffb2452cf8ef09e8f7. Status: Network error: Client connection negotiation failed: client connection to 127.27.122.252:43451: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 16:34:48.730458 2377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } } }
I20260928 16:34:48.730500 2378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } } }
I20260928 16:34:48.730557 2377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:48.730615 2378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:48.730702 2575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:48.730923 2389 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:48.731081 2389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260928 16:34:48.731127 2389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/wal
uuid: "b414e4445ad146ffb2452cf8ef09e8f7"
format_stamp: "Formatted at 2026-09-28 16:34:48 on dist-test-slave-g9sh"
server_key: "fc7829ce584e9bbfd81a63d99a5c66fb"
server_key_iv: "2c3b84342fc30bfcc2128b6c56891bd9"
server_key_version: "encryptionkey@0"
I20260928 16:34:48.731622 2572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } } }
I20260928 16:34:48.731762 2572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:34:48.733767 2389 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:48.736601 2389 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 16:34:48.736698 2389 log.cc:1196] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 16:34:48.736991 2389 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: Deleting consensus metadata
I20260928 16:34:48.737193 2389 master_runner.cc:315] Copying system tablet from 127.27.122.254:41397
I20260928 16:34:48.738454 2389 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: tablet copy: Beginning tablet copy session from remote peer at address 127.27.122.254:41397
I20260928 16:34:48.747464 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.739010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.27.122.254:41397)
0928 16:34:48.739134 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.739139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.739288 (+ 149us) server_negotiation.cc:407] Connection header received
0928 16:34:48.739392 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.739395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.739444 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.739541 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.740042 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.740875 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.742169 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.742534 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.742661 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.744031 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.744035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.744038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.744061 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.745762 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.746225 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.746229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.746231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.746308 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.746607 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.746610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.746612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.746813 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.746882 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.747216 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.747321 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:34:48.747990 2581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.738917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:51728)
0928 16:34:48.739195 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.739209 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.739309 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.739587 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.739590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.739803 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.739918 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.739925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.741061 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.741065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.742049 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.742062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.742779 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.742783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.742882 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.743454 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.743469 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.743889 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.745883 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.745886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.745888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.746119 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.746415 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.746418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.746420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.746498 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.746953 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.746956 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.747043 (+ 87us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.747689 (+ 646us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.747826 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":80,"threads_started":1}
I20260928 16:34:48.748502 1830 tablet_copy_service.cc:140] P 7a8e7bdef6c440389e1041f44c744b40: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b414e4445ad146ffb2452cf8ef09e8f7 ({username='oryx', principal='oryx/127.27.122.252@KRBTEST.COM'} at 127.0.0.1:51728)
I20260928 16:34:48.748597 1830 tablet_copy_service.cc:161] P 7a8e7bdef6c440389e1041f44c744b40: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b414e4445ad146ffb2452cf8ef09e8f7 at {username='oryx', principal='oryx/127.27.122.252@KRBTEST.COM'} at 127.0.0.1:51728: session id = b414e4445ad146ffb2452cf8ef09e8f7-00000000000000000000000000000000
I20260928 16:34:48.749683 1830 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 16:34:48.750891 2389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 16:34:48.753014 2389 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: tablet copy: Starting download of 0 data blocks...
I20260928 16:34:48.753187 2389 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: tablet copy: Starting download of 1 WAL segments...
I20260928 16:34:48.754503 2389 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 16:34:48.756067 1830 tablet_copy_service.cc:342] P 7a8e7bdef6c440389e1041f44c744b40: Request end of tablet copy session b414e4445ad146ffb2452cf8ef09e8f7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.122.252@KRBTEST.COM'} at 127.0.0.1:51728
I20260928 16:34:48.756139 1830 tablet_copy_service.cc:433] P 7a8e7bdef6c440389e1041f44c744b40: ending tablet copy session b414e4445ad146ffb2452cf8ef09e8f7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b414e4445ad146ffb2452cf8ef09e8f7
I20260928 16:34:48.763041 2389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:48.764750 2582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:48.764964 2585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:48.765000 2583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:48.765091 2389 server_base.cc:1034] running on GCE node
I20260928 16:34:48.765376 2389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:48.765681 2389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:48.766837 2389 hybrid_clock.cc:648] HybridClock initialized: now 1790613288766817 us; error 39 us; skew 500 ppm
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613288, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:48.768476 2389 init.cc:377] Logged in from keytab as oryx/127.27.122.252@KRBTEST.COM (short username oryx)
I20260928 16:34:48.769040 2389 webserver.cc:460] Webserver started at http://127.27.122.252:42113/ using document root <none> and password file <none>
I20260928 16:34:48.769255 2389 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:48.769309 2389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:48.770746 2389 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 16:34:48.771853 2592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:48.772130 2389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 16:34:48.772212 2389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/wal
uuid: "b414e4445ad146ffb2452cf8ef09e8f7"
format_stamp: "Formatted at 2026-09-28 16:34:48 on dist-test-slave-g9sh"
server_key: "fc7829ce584e9bbfd81a63d99a5c66fb"
server_key_iv: "2c3b84342fc30bfcc2128b6c56891bd9"
server_key_version: "encryptionkey@0"
I20260928 16:34:48.772305 2389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:48.789315 2389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:48.791664 2389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:48.797546 2389 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.252:43451
I20260928 16:34:48.797583 2644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.252:43451 every 8 connection(s)
I20260928 16:34:48.798266 2389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/master-2/data/info.pb
I20260928 16:34:48.800732 2645 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:34:48.801818 2645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: Bootstrap starting.
I20260928 16:34:48.812114 2645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:34:48.812388 2645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: Bootstrap complete.
I20260928 16:34:48.812922 2645 raft_consensus.cc:348] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } }
I20260928 16:34:48.813053 2645 raft_consensus.cc:729] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b414e4445ad146ffb2452cf8ef09e8f7, State: Initialized, Role: LEARNER
I20260928 16:34:48.813155 2645 consensus_queue.cc:260] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } }
I20260928 16:34:48.813526 2648 sys_catalog.cc:455] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } } }
I20260928 16:34:48.813619 2645 sys_catalog.cc:564] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:48.813712 2648 sys_catalog.cc:458] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: This master's current role is: LEARNER
I20260928 16:34:48.818102 2659 catalog_manager.cc:1259] Loaded cluster ID: 90365853bc0a4c12a1cac893085786d0
I20260928 16:34:48.818222 2659 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: loading cluster ID for follower catalog manager: success
I20260928 16:34:48.820808 2659 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: acquiring CA information for follower catalog manager: success
I20260928 16:34:48.822244 2659 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 16:34:48.823575 2569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:48.823788 2569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:48.823882 2569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:48.827915 2569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:48.827970 2569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:48.828070 2569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:48.833546 2569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab
--principal=oryx/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:45537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=41065
--tserver_master_addrs=127.27.122.254:41397,127.27.122.253:44047,127.27.122.252:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:48.834502 2569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:48.835421 2569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:34:48.839097 2140 heartbeater.cc:503] Master 127.27.122.254:41397 was elected leader, sending a full tablet report...
W20260928 16:34:48.845196 2665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:48.845208 2569 server_base.cc:1034] running on GCE node
W20260928 16:34:48.845201 2667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:48.845438 2664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:48.845831 2569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:48.846356 2569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:48.847519 2569 hybrid_clock.cc:648] HybridClock initialized: now 1790613288847495 us; error 26 us; skew 500 ppm
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613288, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:48.850435 2569 init.cc:377] Logged in from keytab as oryx/127.27.122.193@KRBTEST.COM (short username oryx)
I20260928 16:34:48.851239 2569 webserver.cc:460] Webserver started at http://127.27.122.193:41065/ using document root <none> and password file <none>
I20260928 16:34:48.851720 2569 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:48.851790 2569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:48.855271 2569 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:34:48.857201 2674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:48.858340 2569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:34:48.858472 2569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "4e4a75965fcd4f65af7ed425d7d56296"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "15fe17ae5fd34be4c6b833e912a06a52"
server_key_iv: "0bc15d6fe6fd9525a2cb6df5ee0aa9f5"
server_key_version: "encryptionkey@0"
I20260928 16:34:48.859053 2569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:48.872243 2569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:48.875633 2569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:48.875833 2569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:48.876387 2569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:48.877226 2569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:48.877270 2569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:48.877337 2569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:48.877367 2569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:48.885995 2569 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:45537
I20260928 16:34:48.886063 2789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:45537 every 8 connection(s)
I20260928 16:34:48.886961 2569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:48.890477 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 2569
I20260928 16:34:48.890667 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 2009
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613288, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
I20260928 16:34:48.900267 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:38565
--local_ip_for_outbound_sockets=127.27.122.194
--tserver_master_addrs=127.27.122.254:41397
--webserver_port=43791
--webserver_interface=127.27.122.194
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab
--principal=oryx/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.122.254:41397,127.27.122.253:44047
--tserver_master_addrs=127.27.122.254:41397,127.27.122.253:44047,127.27.122.252:43451 with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613288, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.253@KRBTEST.COM
Sep 28 16:34:48 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613288, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.252@KRBTEST.COM
I20260928 16:34:48.905388 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.888703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:55749 (local address 127.27.122.254:41397)
0928 16:34:48.888837 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.888843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.890573 (+ 1730us) server_negotiation.cc:407] Connection header received
0928 16:34:48.892194 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.892201 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.892265 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.892319 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.894215 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.895052 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.896770 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.897124 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.897195 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.900728 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.900734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.900738 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.900778 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.902897 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.903512 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.903518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.903521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.903620 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.904050 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.904055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.904058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.904298 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.904369 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.905077 (+ 708us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.905218 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:34:48.905455 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.890201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:55777 (local address 127.27.122.253:44047)
0928 16:34:48.890345 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.890350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.891560 (+ 1210us) server_negotiation.cc:407] Connection header received
0928 16:34:48.892930 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.892936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.893003 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.893089 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.894411 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.895188 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.896976 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.897300 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.897373 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.901443 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.901449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.901455 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.901488 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.903399 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.904112 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.904116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.904117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.904193 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.904540 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.904544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.904545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.904748 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.904822 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.905208 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.905318 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:34:48.906402 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.888792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:50619 (local address 127.27.122.252:43451)
0928 16:34:48.889037 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:48.889040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:48.890333 (+ 1293us) server_negotiation.cc:407] Connection header received
0928 16:34:48.892218 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:48.892225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:48.892300 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:48.892378 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:48.894228 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.895436 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.896808 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.897287 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.897379 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.901790 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:48.901796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:48.901802 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:48.901837 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:48.904389 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.904937 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.904944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.904949 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.905044 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:48.905393 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:48.905399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:48.905404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:48.905658 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:48.905740 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:48.906122 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 16:34:48.906246 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
I20260928 16:34:48.906996 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.889427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:44047 (local address 127.27.122.193:55777)
0928 16:34:48.891458 (+ 2031us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.891469 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.892772 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.893484 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.893486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.893648 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.894287 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.894296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.895510 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.895513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.896725 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.896735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.897514 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.897517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.897665 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.898611 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.898629 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.900951 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.903738 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.903740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.903742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.903988 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.904314 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.904319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.904324 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.904436 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.904893 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.904919 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.905016 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.905676 (+ 660us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.906041 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1268,"mutex_wait_us":602,"thread_start_us":94,"threads_started":1}
I20260928 16:34:48.907289 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.889170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:43451 (local address 127.27.122.193:50619)
0928 16:34:48.889748 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.889799 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.891953 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.892709 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.892719 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.893296 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.894045 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.894066 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.895589 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.895605 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.896667 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.896679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.897574 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.897580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.897752 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.898477 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.898498 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.901638 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.904515 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.904521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.904526 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.904781 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.905174 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.905179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.905183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.905280 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.905783 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.905789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.905904 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.906020 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.906193 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":91,"spinlock_wait_cycles":13184,"thread_start_us":689,"threads_started":1}
I20260928 16:34:48.907469 2796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:48.889170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.193:55749)
0928 16:34:48.890475 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:48.890486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:48.891953 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:48.892707 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:48.892719 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:48.893414 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:48.894043 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.894074 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.895236 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.895240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.896635 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:48.896646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:48.897388 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:48.897392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:48.897624 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:48.898320 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:48.898345 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:48.900526 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:48.903027 (+ 2501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.903034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.903050 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.903399 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.903714 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:48.903719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:48.903721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:48.903922 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:48.904449 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:48.904457 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:48.904772 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 16:34:48.905563 (+ 791us) client_negotiation.cc:240] Negotiation successful
0928 16:34:48.907289 (+ 1726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1208,"mutex_wait_us":1511,"thread_start_us":165,"threads_started":1}
I20260928 16:34:48.908510 2792 heartbeater.cc:347] Connected to a master server at 127.27.122.253:44047
I20260928 16:34:48.908510 2791 heartbeater.cc:347] Connected to a master server at 127.27.122.252:43451
I20260928 16:34:48.908785 2792 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:48.908785 2791 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:48.909276 2790 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41397
I20260928 16:34:48.909360 2790 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:48.909420 2791 heartbeater.cc:511] Master 127.27.122.252:43451 requested a full tablet report, sending...
I20260928 16:34:48.909520 2790 heartbeater.cc:511] Master 127.27.122.254:41397 requested a full tablet report, sending...
I20260928 16:34:48.909994 2792 heartbeater.cc:511] Master 127.27.122.253:44047 requested a full tablet report, sending...
I20260928 16:34:48.911051 2424 ts_manager.cc:194] Registered new tserver with Master: 4e4a75965fcd4f65af7ed425d7d56296 (127.27.122.193:45537)
I20260928 16:34:48.911051 2609 ts_manager.cc:194] Registered new tserver with Master: 4e4a75965fcd4f65af7ed425d7d56296 (127.27.122.193:45537)
I20260928 16:34:48.911089 1810 ts_manager.cc:194] Re-registered known tserver with Master: 4e4a75965fcd4f65af7ed425d7d56296 (127.27.122.193:45537)
I20260928 16:34:48.911940 1810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:55749
W20260928 16:34:49.013558 2798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:49.013818 2798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:49.013898 2798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:49.017970 2798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:49.018043 2798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:49.018134 2798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:49.023115 2798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab
--principal=oryx/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:38565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=43791
--tserver_master_addrs=127.27.122.254:41397,127.27.122.253:44047,127.27.122.252:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:49.024068 2798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:49.025040 2798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:49.035393 2806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:49.035401 2803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:49.035483 2804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:49.035744 2798 server_base.cc:1034] running on GCE node
I20260928 16:34:49.036139 2798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:49.036803 2798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:49.037971 2798 hybrid_clock.cc:648] HybridClock initialized: now 1790613289037952 us; error 29 us; skew 500 ppm
Sep 28 16:34:49 dist-test-slave-g9sh krb5kdc[1764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613289, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:49.040517 2798 init.cc:377] Logged in from keytab as oryx/127.27.122.194@KRBTEST.COM (short username oryx)
I20260928 16:34:49.041378 2798 webserver.cc:460] Webserver started at http://127.27.122.194:43791/ using document root <none> and password file <none>
I20260928 16:34:49.041915 2798 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:49.042021 2798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:49.045607 2798 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:49.047205 2275 heartbeater.cc:503] Master 127.27.122.254:41397 was elected leader, sending a full tablet report...
I20260928 16:34:49.047654 2813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:49.048969 2798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 16:34:49.049093 2798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "d4419df5c25243b2999c0bec329e149c"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "a4ac8a45fb147f87b164ee54f6281080"
server_key_iv: "37c1c9c1234e06328a7cea8589ce6ab3"
server_key_version: "encryptionkey@0"
I20260928 16:34:49.049597 2798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:49.065524 2798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:49.068162 2798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:49.068351 2798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:49.068941 2798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:49.069742 2798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:49.069811 2798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:49.069877 2798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:49.069928 2798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:49.078954 2798 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:38565
I20260928 16:34:49.078979 2927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:38565 every 8 connection(s)
I20260928 16:34:49.079896 2798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:49.087581 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 2798
I20260928 16:34:49.087779 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 2145
Sep 28 16:34:49 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613289, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.253@KRBTEST.COM
Sep 28 16:34:49 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613289, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.252@KRBTEST.COM
Sep 28 16:34:49 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613289, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
I20260928 16:34:49.096935 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:44807
--local_ip_for_outbound_sockets=127.27.122.195
--tserver_master_addrs=127.27.122.254:41397
--webserver_port=43523
--webserver_interface=127.27.122.195
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab
--principal=oryx/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:35171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.122.254:41397,127.27.122.253:44047
--tserver_master_addrs=127.27.122.254:41397,127.27.122.253:44047,127.27.122.252:43451 with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:34:49.097577 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.081780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:47423 (local address 127.27.122.252:43451)
0928 16:34:49.081915 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.081919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.082803 (+ 884us) server_negotiation.cc:407] Connection header received
0928 16:34:49.084164 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.084168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.084241 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.084295 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:49.085849 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.086402 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.088625 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.088989 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.089066 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.092534 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:49.092539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:49.092543 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:49.092574 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:49.094965 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.095575 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.095582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.095586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.095693 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.096527 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.096532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.096537 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.096774 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:49.096848 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.097243 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.097409 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:34:49.097680 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.081672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:56687 (local address 127.27.122.253:44047)
0928 16:34:49.081929 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.081933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.083296 (+ 1363us) server_negotiation.cc:407] Connection header received
0928 16:34:49.084114 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.084118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.084171 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.084226 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:49.085848 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.086649 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.088487 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.088937 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.089029 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.092152 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:49.092156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:49.092159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:49.092189 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:49.094712 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.095565 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.095572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.095575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.095677 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.096141 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.096144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.096146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.096409 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:49.096500 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.097264 (+ 764us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.097422 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260928 16:34:49.098215 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.081806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:60567 (local address 127.27.122.254:41397)
0928 16:34:49.082022 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.082026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.083030 (+ 1004us) server_negotiation.cc:407] Connection header received
0928 16:34:49.084111 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.084114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.084171 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.084226 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:49.085848 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.086649 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.088253 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.088580 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.088657 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.093510 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:49.093517 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:49.093522 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:49.093561 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:49.095821 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.096486 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.096492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.096497 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.096593 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.096933 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.096937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.096938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.097170 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:49.097243 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.097927 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.098049 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260928 16:34:49.099220 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.082044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:44047 (local address 127.27.122.194:56687)
0928 16:34:49.083194 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:49.083207 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:49.083853 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:49.084587 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:49.084589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:49.085157 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:49.085704 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.085712 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.086758 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.086762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.088347 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.088362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.089198 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.089204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.089333 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.090060 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:49.090089 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:49.091956 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:49.094903 (+ 2947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:49.094912 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:49.094932 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:49.095423 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:49.095811 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:49.095814 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:49.095816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:49.096019 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:49.096608 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:49.096615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:49.096954 (+ 339us) client_negotiation.cc:769] Sending connection context
0928 16:34:49.097894 (+ 940us) client_negotiation.cc:240] Negotiation successful
0928 16:34:49.098325 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"mutex_wait_us":763,"spinlock_wait_cycles":39808,"thread_start_us":682,"threads_started":1}
I20260928 16:34:49.099228 2932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.082044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:43451 (local address 127.27.122.194:47423)
0928 16:34:49.082624 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:49.082686 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:49.083897 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:49.084577 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:49.084583 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:49.085154 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:49.085687 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.085702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.086552 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.086555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.088485 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.088494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.089238 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.089242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.089386 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.089985 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:49.090016 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:49.092361 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:49.095093 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:49.095096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:49.095098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:49.095458 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:49.095818 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:49.095823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:49.095828 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:49.096025 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:49.096940 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:49.096944 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:49.097036 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 16:34:49.097837 (+ 801us) client_negotiation.cc:240] Negotiation successful
0928 16:34:49.098210 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":96,"thread_start_us":170,"threads_started":1}
I20260928 16:34:49.099228 2933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.082044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.194:60567)
0928 16:34:49.082953 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:49.082965 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:49.083853 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:49.084580 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:49.084585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:49.085157 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:49.085702 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.085712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.087257 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.087260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.088123 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.088133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.088844 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.088848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.089080 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.090061 (+ 981us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:49.090080 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:49.093343 (+ 3263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:49.096113 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:49.096118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:49.096123 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:49.096394 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:49.096706 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:49.096709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:49.096711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:49.096794 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:49.097461 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:49.097464 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:49.097549 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 16:34:49.098444 (+ 895us) client_negotiation.cc:240] Negotiation successful
0928 16:34:49.098576 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":105,"spinlock_wait_cycles":58368,"thread_start_us":135,"threads_started":1}
I20260928 16:34:49.100531 2930 heartbeater.cc:347] Connected to a master server at 127.27.122.253:44047
I20260928 16:34:49.100528 2928 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41397
I20260928 16:34:49.100533 2929 heartbeater.cc:347] Connected to a master server at 127.27.122.252:43451
I20260928 16:34:49.100809 2930 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:49.100808 2928 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:49.100808 2929 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:49.101471 2929 heartbeater.cc:511] Master 127.27.122.252:43451 requested a full tablet report, sending...
I20260928 16:34:49.101471 2928 heartbeater.cc:511] Master 127.27.122.254:41397 requested a full tablet report, sending...
I20260928 16:34:49.101490 2930 heartbeater.cc:511] Master 127.27.122.253:44047 requested a full tablet report, sending...
I20260928 16:34:49.102797 2609 ts_manager.cc:194] Registered new tserver with Master: d4419df5c25243b2999c0bec329e149c (127.27.122.194:38565)
I20260928 16:34:49.102836 1810 ts_manager.cc:194] Re-registered known tserver with Master: d4419df5c25243b2999c0bec329e149c (127.27.122.194:38565)
I20260928 16:34:49.102867 2424 ts_manager.cc:194] Registered new tserver with Master: d4419df5c25243b2999c0bec329e149c (127.27.122.194:38565)
I20260928 16:34:49.103511 1810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:60567
I20260928 16:34:49.172679 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.168713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:43451 (local address 127.0.0.1:32822)
0928 16:34:49.168855 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:49.168873 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:49.169031 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:49.169356 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:49.169358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:49.169374 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:49.169472 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.169478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.170518 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.170520 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.171324 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.171334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.172302 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.172306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.172401 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.172425 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:34:49.172485 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 16:34:49.172552 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 16:34:49.172755 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.168794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32822 (local address 127.27.122.252:43451)
0928 16:34:49.168972 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.168977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.168998 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:34:49.169145 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.169149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.169218 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.169338 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:49.169583 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.170394 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.171450 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.172168 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.172283 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.172412 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.172525 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.172595 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:34:49.173429 2619 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 16:34:49.173799 2648 sys_catalog.cc:455] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a8e7bdef6c440389e1041f44c744b40. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } } }
I20260928 16:34:49.173974 2648 sys_catalog.cc:458] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: This master's current role is: LEARNER
W20260928 16:34:49.211978 2935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:49.212200 2935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:49.212239 2935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:49.216378 2935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:49.216429 2935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:49.216536 2935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:49.221805 2935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab
--principal=oryx/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:44807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=43523
--tserver_master_addrs=127.27.122.254:41397,127.27.122.253:44047,127.27.122.252:43451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:49.222735 2935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:49.223691 2935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:49.233026 2941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:49.233047 2940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:49.233067 2943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:49.233548 2935 server_base.cc:1034] running on GCE node
I20260928 16:34:49.233922 2935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:49.234494 2935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:49.235680 2935 hybrid_clock.cc:648] HybridClock initialized: now 1790613289235651 us; error 46 us; skew 500 ppm
Sep 28 16:34:49 dist-test-slave-g9sh krb5kdc[1764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613289, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:49.238634 2935 init.cc:377] Logged in from keytab as oryx/127.27.122.195@KRBTEST.COM (short username oryx)
I20260928 16:34:49.239498 2935 webserver.cc:460] Webserver started at http://127.27.122.195:43523/ using document root <none> and password file <none>
I20260928 16:34:49.240033 2935 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:49.240160 2935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:49.243829 2935 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:49.245771 2950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:49.246918 2935 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:49.247047 2935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "5910840c736d43018174009dfc1218f6"
format_stamp: "Formatted at 2026-09-28 16:34:47 on dist-test-slave-g9sh"
server_key: "fcbf0aa0762d394dfa75d6e1e53b763f"
server_key_iv: "d023071dfcb4e080dc9284ee9cf0cb43"
server_key_version: "encryptionkey@0"
I20260928 16:34:49.247555 2935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:49.269879 2935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:49.273440 2935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:49.273669 2935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:49.274299 2935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:49.275184 2935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:49.275229 2935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:49.275270 2935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:49.275324 2935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:49.283958 2935 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:44807
I20260928 16:34:49.283984 3066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:44807 every 8 connection(s)
I20260928 16:34:49.285051 2935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:34:49.286039 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 2935
Sep 28 16:34:49 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613289, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
Sep 28 16:34:49 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613289, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.252@KRBTEST.COM
Sep 28 16:34:49 dist-test-slave-g9sh krb5kdc[1764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613289, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.253@KRBTEST.COM
I20260928 16:34:49.301328 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.287998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.27.122.253:44047)
0928 16:34:49.288500 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.288504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.289324 (+ 820us) server_negotiation.cc:407] Connection header received
0928 16:34:49.289482 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.289485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.289540 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.289612 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:49.290743 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.291615 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.293056 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.293417 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.293498 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.296136 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:49.296140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:49.296143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:49.296169 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:49.298785 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.299325 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.299330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.299333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.299424 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.299717 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.299774 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.299780 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.300003 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:49.300144 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.300781 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.301152 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":83,"threads_started":1}
I20260928 16:34:49.304569 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.287790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:58199 (local address 127.27.122.253:44047)
0928 16:34:49.288369 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.288376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.288802 (+ 426us) server_negotiation.cc:407] Connection header received
0928 16:34:49.289983 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.289989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.290033 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.290102 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:49.294236 (+ 4134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.294951 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.297175 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.297524 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.297606 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.300248 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:49.300253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:49.300259 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:49.300293 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:49.302663 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.303165 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.303168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.303170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.303259 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.303567 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.303570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.303572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.303763 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:49.303824 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.304303 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.304424 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479}
I20260928 16:34:49.304672 1849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.286901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:32963 (local address 127.27.122.254:41397)
0928 16:34:49.287595 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.287600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.288645 (+ 1045us) server_negotiation.cc:407] Connection header received
0928 16:34:49.290091 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.290095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.290155 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.290241 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:49.292048 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.292829 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.294655 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.294977 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.295054 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.298972 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:49.298978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:49.298983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:49.299019 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:49.302200 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.302820 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.302824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.302827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.302920 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.303374 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.303378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.303379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.303610 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:49.303697 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.304343 (+ 646us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.304473 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590}
I20260928 16:34:49.305891 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.287343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:44047 (local address 127.27.122.195:58199)
0928 16:34:49.288712 (+ 1369us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:49.288726 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:49.289753 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:49.290426 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:49.290435 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:49.291103 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:49.294050 (+ 2947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.294057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.296161 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.296166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.297009 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.297023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.297792 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.297795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.297903 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.298503 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:49.298519 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:49.300076 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:49.302787 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:49.302790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:49.302792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:49.303026 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:49.303363 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:49.303366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:49.303368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:49.303452 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:49.303872 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:49.303874 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:49.304025 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 16:34:49.304250 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 16:34:49.304568 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1284,"mutex_wait_us":396,"spinlock_wait_cycles":68608,"thread_start_us":148,"threads_started":1}
I20260928 16:34:49.306914 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.287343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.195:32963)
0928 16:34:49.288490 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:49.288550 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:49.289767 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:49.290428 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:49.290436 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:49.291095 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:49.291886 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.291905 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.293170 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.293174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.294511 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.294521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.295253 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.295257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.295462 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.296106 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:49.296129 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:49.298824 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:49.302335 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:49.302341 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:49.302357 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:49.302695 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:49.303045 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:49.303048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:49.303051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:49.303252 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:49.303769 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:49.303777 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:49.304033 (+ 256us) client_negotiation.cc:769] Sending connection context
0928 16:34:49.304797 (+ 764us) client_negotiation.cc:240] Negotiation successful
0928 16:34:49.304976 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":144,"spinlock_wait_cycles":7552,"thread_start_us":149,"threads_started":1}
I20260928 16:34:49.307617 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.288891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:45617 (local address 127.27.122.252:43451)
0928 16:34:49.289087 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.289092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.289646 (+ 554us) server_negotiation.cc:407] Connection header received
0928 16:34:49.290146 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.290149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.290209 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.290263 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:49.292052 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.292854 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.294646 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.294987 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.295062 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.301080 (+ 6018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:49.301086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:49.301089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:49.301129 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:49.303875 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.305516 (+ 1641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.305520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.305521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.305597 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.306041 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.306043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.306045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.306294 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:49.306398 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.307316 (+ 918us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.307437 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:34:49.307672 3067 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41397
I20260928 16:34:49.307752 3069 heartbeater.cc:347] Connected to a master server at 127.27.122.253:44047
I20260928 16:34:49.307889 3067 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:49.307889 3069 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:49.308187 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.289032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.27.122.252:43451)
0928 16:34:49.289225 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.289239 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.291075 (+ 1836us) server_negotiation.cc:407] Connection header received
0928 16:34:49.291124 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.291127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.291182 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.291269 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:49.292192 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.295057 (+ 2865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.297589 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.297966 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.298072 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.299704 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:49.299708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:49.299711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:49.299736 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:49.305516 (+ 5780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.306011 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.306017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.306022 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.306446 (+ 424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.306777 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.306780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.306782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.307028 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:49.307153 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.307852 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.308013 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":85,"threads_started":1}
I20260928 16:34:49.308506 3067 heartbeater.cc:511] Master 127.27.122.254:41397 requested a full tablet report, sending...
I20260928 16:34:49.308506 3069 heartbeater.cc:511] Master 127.27.122.253:44047 requested a full tablet report, sending...
I20260928 16:34:49.309585 3075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.288447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:43451 (local address 127.27.122.195:45617)
0928 16:34:49.289555 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:49.289568 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:49.290025 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:49.290426 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:49.290435 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:49.291193 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:49.291888 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.291909 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.293030 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.293034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.294514 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.294524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.295253 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.295256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.295459 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.296494 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:49.296510 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:49.298544 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:49.305121 (+ 6577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:49.305126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:49.305131 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:49.305391 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:49.305707 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:49.305712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:49.305717 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:49.305940 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:49.306411 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:49.306416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:49.306524 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 16:34:49.309290 (+ 2766us) client_negotiation.cc:240] Negotiation successful
0928 16:34:49.309427 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":136,"spinlock_wait_cycles":16384,"thread_start_us":135,"threads_started":1}
I20260928 16:34:49.309617 1810 ts_manager.cc:194] Re-registered known tserver with Master: 5910840c736d43018174009dfc1218f6 (127.27.122.195:44807)
I20260928 16:34:49.309692 2424 ts_manager.cc:194] Registered new tserver with Master: 5910840c736d43018174009dfc1218f6 (127.27.122.195:44807)
I20260928 16:34:49.310081 3068 heartbeater.cc:347] Connected to a master server at 127.27.122.252:43451
I20260928 16:34:49.310194 3068 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:49.310359 3068 heartbeater.cc:511] Master 127.27.122.252:43451 requested a full tablet report, sending...
I20260928 16:34:49.310339 1810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:32963
I20260928 16:34:49.311048 2609 ts_manager.cc:194] Registered new tserver with Master: 5910840c736d43018174009dfc1218f6 (127.27.122.195:44807)
I20260928 16:34:49.312620 3082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.289473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51730 (local address 127.27.122.254:41397)
0928 16:34:49.299994 (+ 10521us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.299999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.300014 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:34:49.300052 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.300056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.300106 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.300915 (+ 809us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:49.301028 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.301790 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.302788 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.303173 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.303622 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.305180 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:49.305183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:49.305186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:49.305210 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:49.310840 (+ 5630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.311480 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.311484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.311485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.311584 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.311827 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.311830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.311832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.312026 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:49.312150 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.312314 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.312405 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9086,"thread_start_us":83,"threads_started":1}
I20260928 16:34:49.315433 1810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:34:49.318516 1810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:49.320878 2376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: NON_VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: true } } }
I20260928 16:34:49.321111 2376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:49.326470 2376 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40: attempting to promote NON_VOTER b414e4445ad146ffb2452cf8ef09e8f7 to VOTER
I20260928 16:34:49.327092 2376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } }
I20260928 16:34:49.328495 2619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 LEARNER]: Refusing update from remote peer 7a8e7bdef6c440389e1041f44c744b40: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260928 16:34:49.328887 3086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 16:34:49.329490 2433 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 FOLLOWER]: Refusing update from remote peer 7a8e7bdef6c440389e1041f44c744b40: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260928 16:34:49.329879 3086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 16:34:49.331722 2434 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20260928 16:34:49.332407 3085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, b414e4445ad146ffb2452cf8ef09e8f7 (127.27.122.252) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } } }
I20260928 16:34:49.332793 2434 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, b414e4445ad146ffb2452cf8ef09e8f7 (127.27.122.252) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } } }
I20260928 16:34:49.334486 3085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } } }
I20260928 16:34:49.334602 3085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:49.334671 3086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } } }
I20260928 16:34:49.334826 3086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:49.334812 3085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } } }
I20260928 16:34:49.334908 3085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:49.334580 3094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } } }
I20260928 16:34:49.335037 3094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:34:49.334769 2619 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, b414e4445ad146ffb2452cf8ef09e8f7 (127.27.122.252) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } } }
I20260928 16:34:49.336606 2648 sys_catalog.cc:455] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a8e7bdef6c440389e1041f44c744b40" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } } }
I20260928 16:34:49.336721 2648 sys_catalog.cc:458] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:34:49.341398 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.336410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:45537 (local address 127.0.0.1:54236)
0928 16:34:49.336885 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:49.336925 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:49.337008 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:49.337560 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:49.337563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:49.337574 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:49.337662 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.337666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.339023 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.339025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.339832 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.339841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.340973 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.340977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.341067 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.341095 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:34:49.341191 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 16:34:49.341277 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403}
I20260928 16:34:49.341470 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.336572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54236 (local address 127.27.122.193:45537)
0928 16:34:49.337057 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.337070 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.337110 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:34:49.337203 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.337208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.337425 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.337576 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:49.337739 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.338924 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.339955 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.340808 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.340959 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.341119 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.341222 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.341327 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":104,"threads_started":1}
I20260928 16:34:49.344499 2723 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae10a62e2fb4760b085e1703cf0ef1d (DEFAULT_TABLE table=test-table [id=303ff9c971104248b6afaddf910604d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:49.345614 2723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae10a62e2fb4760b085e1703cf0ef1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:49.350203 3097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.336688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:38565 (local address 127.0.0.1:38482)
0928 16:34:49.337349 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:49.337362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:49.337432 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:49.345597 (+ 8165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:49.345601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:49.345620 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:49.345791 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.345799 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.347296 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.347299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.348482 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.348491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.349776 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.349779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.349905 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.349928 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:34:49.349993 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 16:34:49.350065 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":116,"threads_started":1}
I20260928 16:34:49.350888 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.341075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.27.122.194:38565)
0928 16:34:49.345217 (+ 4142us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.345226 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.345245 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:34:49.345307 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.345310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.345454 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.345618 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:49.345930 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.347165 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.348631 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.349479 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.349596 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.349713 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.350623 (+ 910us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.350726 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4052,"thread_start_us":111,"threads_started":1}
I20260928 16:34:49.351456 3098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.337047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:44807 (local address 127.0.0.1:35406)
0928 16:34:49.337300 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:49.337314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:49.337405 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:49.346485 (+ 9080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:49.346489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:49.346507 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:49.346591 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.346608 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.348631 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.348634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.349772 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.349780 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.351061 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.351067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.351160 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.351189 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:34:49.351263 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 16:34:49.351338 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":119,"threads_started":1}
I20260928 16:34:49.351979 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.341140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.27.122.195:44807)
0928 16:34:49.345927 (+ 4787us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.345939 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.345959 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:34:49.346032 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.346037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.346211 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.346352 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:49.346727 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.348493 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.349913 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.350738 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.350852 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.350991 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.351721 (+ 730us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.351819 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4671,"thread_start_us":101,"threads_started":1}
I20260928 16:34:49.353638 3101 tablet_bootstrap.cc:492] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296: Bootstrap starting.
I20260928 16:34:49.353946 2862 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae10a62e2fb4760b085e1703cf0ef1d (DEFAULT_TABLE table=test-table [id=303ff9c971104248b6afaddf910604d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:49.354768 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae10a62e2fb4760b085e1703cf0ef1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:49.355711 3101 tablet_bootstrap.cc:654] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:49.356419 3101 log.cc:824] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:49.359469 3000 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae10a62e2fb4760b085e1703cf0ef1d (DEFAULT_TABLE table=test-table [id=303ff9c971104248b6afaddf910604d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:49.360414 3101 tablet_bootstrap.cc:492] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296: No bootstrap required, opened a new log
I20260928 16:34:49.360509 3000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae10a62e2fb4760b085e1703cf0ef1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:49.360666 3101 ts_tablet_manager.cc:1397] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:34:49.361799 3103 tablet_bootstrap.cc:492] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c: Bootstrap starting.
I20260928 16:34:49.363817 3103 tablet_bootstrap.cc:654] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:49.364491 3103 log.cc:824] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:49.364874 3101 raft_consensus.cc:348] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } } peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } }
I20260928 16:34:49.365185 3101 raft_consensus.cc:374] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:49.365247 3101 raft_consensus.cc:729] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a75965fcd4f65af7ed425d7d56296, State: Initialized, Role: FOLLOWER
I20260928 16:34:49.365700 3101 consensus_queue.cc:260] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } } peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } }
I20260928 16:34:49.366364 2790 heartbeater.cc:503] Master 127.27.122.254:41397 was elected leader, sending a full tablet report...
I20260928 16:34:49.366386 3103 tablet_bootstrap.cc:492] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c: No bootstrap required, opened a new log
I20260928 16:34:49.366547 3103 ts_tablet_manager.cc:1397] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 16:34:49.366711 3101 ts_tablet_manager.cc:1428] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260928 16:34:49.369877 3106 tablet_bootstrap.cc:492] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6: Bootstrap starting.
I20260928 16:34:49.370929 3103 raft_consensus.cc:348] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } } peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } }
I20260928 16:34:49.371213 3103 raft_consensus.cc:374] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:49.371284 3103 raft_consensus.cc:729] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4419df5c25243b2999c0bec329e149c, State: Initialized, Role: FOLLOWER
I20260928 16:34:49.371727 3106 tablet_bootstrap.cc:654] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:49.371726 3103 consensus_queue.cc:260] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } } peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } }
I20260928 16:34:49.372413 2928 heartbeater.cc:503] Master 127.27.122.254:41397 was elected leader, sending a full tablet report...
I20260928 16:34:49.373898 3106 log.cc:824] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:49.375095 3103 ts_tablet_manager.cc:1428] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260928 16:34:49.375941 3106 tablet_bootstrap.cc:492] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6: No bootstrap required, opened a new log
I20260928 16:34:49.376138 3106 ts_tablet_manager.cc:1397] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 16:34:49.379808 3106 raft_consensus.cc:348] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } } peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } }
I20260928 16:34:49.380028 3106 raft_consensus.cc:374] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:49.380086 3106 raft_consensus.cc:729] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5910840c736d43018174009dfc1218f6, State: Initialized, Role: FOLLOWER
I20260928 16:34:49.380652 3106 consensus_queue.cc:260] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } } peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } }
I20260928 16:34:49.381325 3067 heartbeater.cc:503] Master 127.27.122.254:41397 was elected leader, sending a full tablet report...
I20260928 16:34:49.381608 3106 ts_tablet_manager.cc:1428] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260928 16:34:49.388792 2793 tablet.cc:2369] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:49.519632 3105 raft_consensus.cc:482] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:49.519861 3105 raft_consensus.cc:504] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } } peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } }
I20260928 16:34:49.521416 3105 leader_election.cc:288] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4419df5c25243b2999c0bec329e149c (127.27.122.194:38565), 5910840c736d43018174009dfc1218f6 (127.27.122.195:44807)
I20260928 16:34:49.526726 3100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.521836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:49487 (local address 127.27.122.195:44807)
0928 16:34:49.521994 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.522001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.522076 (+ 75us) server_negotiation.cc:407] Connection header received
0928 16:34:49.522252 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.522255 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.522299 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.522386 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:49.522749 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.523539 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.525372 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.526008 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.526139 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.526192 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.526514 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.526594 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 16:34:49.526726 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.521767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:56539 (local address 127.27.122.194:38565)
0928 16:34:49.521967 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.521972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.522052 (+ 80us) server_negotiation.cc:407] Connection header received
0928 16:34:49.522242 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.522245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.522299 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.522386 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:49.522749 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.523539 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.525372 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.526008 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.526139 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.526192 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.526515 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.526605 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 16:34:49.526755 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.521629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:38565 (local address 127.27.122.193:56539)
0928 16:34:49.521965 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:49.521981 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:49.522101 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:49.522416 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:49.522419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:49.522448 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:49.522636 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.522641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.523656 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.523660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.525234 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.525246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.526182 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.526187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.526343 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.526416 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 16:34:49.526486 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 16:34:49.526570 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":329,"spinlock_wait_cycles":6912,"thread_start_us":100,"threads_started":1}
I20260928 16:34:49.526765 3112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.521629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:44807 (local address 127.27.122.193:49487)
0928 16:34:49.521974 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:49.521986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:49.522103 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:49.522416 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:49.522419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:49.522447 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:49.522620 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.522629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.523656 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.523660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.525234 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:49.525246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.526182 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.526187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:49.526348 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.526372 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:34:49.526468 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 16:34:49.526570 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":313,"spinlock_wait_cycles":7936,"thread_start_us":65,"threads_started":1}
I20260928 16:34:49.527896 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae10a62e2fb4760b085e1703cf0ef1d" candidate_uuid: "4e4a75965fcd4f65af7ed425d7d56296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4419df5c25243b2999c0bec329e149c" is_pre_election: true
I20260928 16:34:49.527863 3020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae10a62e2fb4760b085e1703cf0ef1d" candidate_uuid: "4e4a75965fcd4f65af7ed425d7d56296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5910840c736d43018174009dfc1218f6" is_pre_election: true
I20260928 16:34:49.528210 3020 raft_consensus.cc:2454] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a75965fcd4f65af7ed425d7d56296 in term 0.
I20260928 16:34:49.528210 2882 raft_consensus.cc:2454] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a75965fcd4f65af7ed425d7d56296 in term 0.
I20260928 16:34:49.528770 2676 leader_election.cc:302] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4e4a75965fcd4f65af7ed425d7d56296, 5910840c736d43018174009dfc1218f6, d4419df5c25243b2999c0bec329e149c; no voters:
I20260928 16:34:49.529147 3105 raft_consensus.cc:2781] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:49.529233 3105 raft_consensus.cc:482] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:49.529264 3105 raft_consensus.cc:3037] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:49.530323 3105 raft_consensus.cc:504] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } } peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } }
I20260928 16:34:49.530678 3105 leader_election.cc:288] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [CANDIDATE]: Term 1 election: Requested vote from peers d4419df5c25243b2999c0bec329e149c (127.27.122.194:38565), 5910840c736d43018174009dfc1218f6 (127.27.122.195:44807)
I20260928 16:34:49.531073 3020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae10a62e2fb4760b085e1703cf0ef1d" candidate_uuid: "4e4a75965fcd4f65af7ed425d7d56296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5910840c736d43018174009dfc1218f6"
I20260928 16:34:49.531170 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae10a62e2fb4760b085e1703cf0ef1d" candidate_uuid: "4e4a75965fcd4f65af7ed425d7d56296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4419df5c25243b2999c0bec329e149c"
I20260928 16:34:49.531205 3020 raft_consensus.cc:3037] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:49.531297 2882 raft_consensus.cc:3037] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:49.532569 2882 raft_consensus.cc:2454] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a75965fcd4f65af7ed425d7d56296 in term 1.
I20260928 16:34:49.532569 3020 raft_consensus.cc:2454] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a75965fcd4f65af7ed425d7d56296 in term 1.
I20260928 16:34:49.532843 2676 leader_election.cc:302] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a75965fcd4f65af7ed425d7d56296, 5910840c736d43018174009dfc1218f6; no voters:
I20260928 16:34:49.533032 3105 raft_consensus.cc:2781] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:49.533216 3105 raft_consensus.cc:686] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4a75965fcd4f65af7ed425d7d56296, State: Running, Role: LEADER
I20260928 16:34:49.533562 3105 consensus_queue.cc:237] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } } peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } }
W20260928 16:34:49.536185 3070 tablet.cc:2369] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:49.536597 1810 catalog_manager.cc:5495] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4a75965fcd4f65af7ed425d7d56296 (127.27.122.193). New cstate: current_term: 1 leader_uuid: "4e4a75965fcd4f65af7ed425d7d56296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } health_report { overall_health: UNKNOWN } } }
I20260928 16:34:49.575047 3096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.571144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53848 (local address 127.27.122.193:45537)
0928 16:34:49.571286 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.571291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.571305 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:34:49.571441 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.571444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.571498 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.571609 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:49.571885 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.572417 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.573501 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.573728 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.573856 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.574242 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:49.574672 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:49.574808 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.574848 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.574904 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:34:49.580751 3020 raft_consensus.cc:1261] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a75965fcd4f65af7ed425d7d56296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:49.580751 2882 raft_consensus.cc:1261] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c [term 1 FOLLOWER]: Refusing update from remote peer 4e4a75965fcd4f65af7ed425d7d56296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:49.581526 3113 consensus_queue.cc:1035] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:49.581738 3105 consensus_queue.cc:1035] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:34:49.581838 2931 tablet.cc:2369] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:49.589716 3116 mvcc.cc:205] Tried to move back new op lower bound from 7334352034115837952 to 7334352033932328960. Current Snapshot: MvccSnapshot[applied={T|T < 7334352034115837952}]
I20260928 16:34:49.592428 3117 mvcc.cc:205] Tried to move back new op lower bound from 7334352034115837952 to 7334352033932328960. Current Snapshot: MvccSnapshot[applied={T|T < 7334352034115837952 or (T in {7334352034115837952})}]
I20260928 16:34:49.600277 1810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:49.600442 1810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:49.604880 1810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a8e7bdef6c440389e1041f44c744b40: Sending DeleteTablet for 3 replicas of tablet 6ae10a62e2fb4760b085e1703cf0ef1d
I20260928 16:34:49.605581 2723 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ae10a62e2fb4760b085e1703cf0ef1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:49 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:54236
I20260928 16:34:49.605850 3000 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ae10a62e2fb4760b085e1703cf0ef1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:49 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:35406
I20260928 16:34:49.605934 2862 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ae10a62e2fb4760b085e1703cf0ef1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:49 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:38482
I20260928 16:34:49.606348 3125 tablet_replica.cc:330] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296: stopping tablet replica
I20260928 16:34:49.606655 3125 raft_consensus.cc:2229] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:49.606774 3127 tablet_replica.cc:330] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c: stopping tablet replica
I20260928 16:34:49.606981 3125 raft_consensus.cc:2258] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:49.607071 3127 raft_consensus.cc:2229] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:49.607398 3127 raft_consensus.cc:2258] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:49.608385 3125 ts_tablet_manager.cc:1905] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:49.608798 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 1777
I20260928 16:34:49.611544 3125 ts_tablet_manager.cc:1918] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:49.611649 3125 log.cc:1196] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-0/wal/wals/6ae10a62e2fb4760b085e1703cf0ef1d
I20260928 16:34:49.611964 3125 ts_tablet_manager.cc:1939] T 6ae10a62e2fb4760b085e1703cf0ef1d P 4e4a75965fcd4f65af7ed425d7d56296: Deleting consensus metadata
I20260928 16:34:49.617136 3126 tablet_replica.cc:330] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6: stopping tablet replica
I20260928 16:34:49.617506 3126 raft_consensus.cc:2229] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260928 16:34:49.617502 2678 connection.cc:535] server connection from 127.0.0.1:54236 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 16:34:49.618913 3126 raft_consensus.cc:2258] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:49.619643 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.607380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.27.122.253:44047)
0928 16:34:49.607616 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.607621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.608418 (+ 797us) server_negotiation.cc:407] Connection header received
0928 16:34:49.608689 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.608692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.608741 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.608794 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:49.610780 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.611691 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.613369 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.613726 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.613890 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.615777 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:49.615781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:49.615784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:49.615809 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:49.617527 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.618218 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.618224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.618229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.618327 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.618664 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.618668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.618670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.618887 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:49.619009 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.619359 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.619474 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260928 16:34:49.620937 3126 ts_tablet_manager.cc:1905] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:49.621558 3127 ts_tablet_manager.cc:1905] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:49.623737 3126 ts_tablet_manager.cc:1918] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:49.623884 3126 log.cc:1196] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/6ae10a62e2fb4760b085e1703cf0ef1d
I20260928 16:34:49.624294 3126 ts_tablet_manager.cc:1939] T 6ae10a62e2fb4760b085e1703cf0ef1d P 5910840c736d43018174009dfc1218f6: Deleting consensus metadata
I20260928 16:34:49.624281 3127 ts_tablet_manager.cc:1918] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:49.624372 3127 log.cc:1196] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/6ae10a62e2fb4760b085e1703cf0ef1d
I20260928 16:34:49.624703 3127 ts_tablet_manager.cc:1939] T 6ae10a62e2fb4760b085e1703cf0ef1d P d4419df5c25243b2999c0bec329e149c: Deleting consensus metadata
W20260928 16:34:49.625617 2954 connection.cc:441] server connection from 127.0.0.1:35406 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35406 (request call id 1) could send its response
W20260928 16:34:49.625804 2814 connection.cc:441] server connection from 127.0.0.1:38482 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38482 (request call id 1) could send its response
I20260928 16:34:49.629729 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.607479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.27.122.252:43451)
0928 16:34:49.607714 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.607720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.620858 (+ 13138us) server_negotiation.cc:407] Connection header received
0928 16:34:49.621067 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.621070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.621126 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.621179 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:49.622019 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.622816 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.623928 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.624281 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.624353 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.626085 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:49.626099 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:49.626104 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:49.626133 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:49.627869 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.628382 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.628386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.628388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.628473 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.628772 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.628775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.628777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.629012 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:49.629112 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.629441 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.629563 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260928 16:34:49.637494 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.626222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.27.122.252:43451)
0928 16:34:49.626323 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.626327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.627546 (+ 1219us) server_negotiation.cc:407] Connection header received
0928 16:34:49.627738 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.627742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.627792 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.627842 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:49.629154 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.629916 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.631617 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.631954 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.632057 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.633806 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:49.633810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:49.633813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:49.633839 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:49.635638 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.636101 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.636107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.636110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.636202 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.636511 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.636515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.636516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.636720 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:49.636814 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.637208 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.637321 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 16:34:49.639086 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:49.626948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.27.122.253:44047)
0928 16:34:49.627056 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:49.627059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:49.630523 (+ 3464us) server_negotiation.cc:407] Connection header received
0928 16:34:49.630689 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:49.630692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:49.630734 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:49.630781 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:49.631752 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.632400 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.633858 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:49.634173 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:49.634297 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:49.635606 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:49.635610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:49.635613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:49.635644 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:49.637302 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.637815 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.637819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.637821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.637907 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:49.638213 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:49.638216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:49.638218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:49.638431 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:49.638553 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:49.638826 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 16:34:49.638936 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:34:50.594605 3136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:50.594133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.194:47289)
0928 16:34:50.594395 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:50.594503 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
W20260928 16:34:50.595017 2928 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:41397 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
I20260928 16:34:50.595754 3137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:50.595371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.193:42397)
0928 16:34:50.595584 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:50.595677 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
W20260928 16:34:50.596037 2790 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:41397 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
I20260928 16:34:50.599259 3138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:50.598728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.195:60183)
0928 16:34:50.599020 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:50.599150 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
W20260928 16:34:50.599673 3067 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:41397 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
I20260928 16:34:51.104624 3139 raft_consensus.cc:482] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8e7bdef6c440389e1041f44c744b40)
I20260928 16:34:51.104882 3139 raft_consensus.cc:504] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } }
I20260928 16:34:51.105973 3139 leader_election.cc:288] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8e7bdef6c440389e1041f44c744b40 (127.27.122.254:41397), 1acf860775654f52a463c0180f8c44b8 (127.27.122.253:44047)
I20260928 16:34:51.106168 3140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7a8e7bdef6c440389e1041f44c744b40)
I20260928 16:34:51.106351 3140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } }
I20260928 16:34:51.106528 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.106058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50362)
0928 16:34:51.106326 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:51.106413 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":61,"threads_started":1}
W20260928 16:34:51.106734 2593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
I20260928 16:34:51.107635 3143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.107365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50376)
0928 16:34:51.107488 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:51.107550 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 16:34:51.107766 2593 leader_election.cc:334] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a8e7bdef6c440389e1041f44c744b40 (127.27.122.254:41397): Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
I20260928 16:34:51.108219 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.107825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50390)
0928 16:34:51.108048 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:51.108125 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
I20260928 16:34:51.108229 3140 leader_election.cc:288] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a8e7bdef6c440389e1041f44c744b40 (127.27.122.254:41397), b414e4445ad146ffb2452cf8ef09e8f7 (127.27.122.252:43451)
W20260928 16:34:51.108546 2408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
I20260928 16:34:51.109546 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.109339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50394)
0928 16:34:51.109442 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:51.109495 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 16:34:51.109670 2408 leader_election.cc:334] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a8e7bdef6c440389e1041f44c744b40 (127.27.122.254:41397): Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
I20260928 16:34:51.111241 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.106479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:44047 (local address 127.0.0.1:53516)
0928 16:34:51.106677 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:51.106691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:51.106795 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:51.107505 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:51.107508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:51.107526 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:51.107631 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:51.107636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:51.108996 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:51.109002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:51.110165 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:51.110173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:51.110828 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:51.110832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:51.110931 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:51.110950 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 16:34:51.111039 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:34:51.111121 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
I20260928 16:34:51.111263 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.106262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.27.122.253:44047)
0928 16:34:51.107221 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:51.107227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:51.107243 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:34:51.107299 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:51.107302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:51.107359 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:51.107444 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:51.108004 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:51.108859 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:51.110285 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:51.110709 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:51.110834 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:51.110884 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:51.111053 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 16:34:51.111135 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":69,"threads_started":1}
I20260928 16:34:51.111699 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1acf860775654f52a463c0180f8c44b8" is_pre_election: true
I20260928 16:34:51.111903 2434 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b414e4445ad146ffb2452cf8ef09e8f7 in term 1.
I20260928 16:34:51.112000 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.108058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:43451 (local address 127.0.0.1:55878)
0928 16:34:51.108450 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:51.108464 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:51.108565 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:51.108830 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:51.108834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:51.108854 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:51.108982 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:51.108990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:51.109962 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:51.109965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:51.110757 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:51.110766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:51.111577 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:51.111581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:51.111674 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:51.111692 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 16:34:51.111780 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:34:51.111852 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":74,"threads_started":1}
I20260928 16:34:51.111999 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.108208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.27.122.252:43451)
0928 16:34:51.108442 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:51.108450 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:51.108519 (+ 69us) server_negotiation.cc:407] Connection header received
0928 16:34:51.108669 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:51.108672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:51.108710 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:51.108790 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:51.109108 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:51.109855 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:51.110887 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:51.111355 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:51.111474 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:51.111516 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:51.111809 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 16:34:51.111880 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":106,"threads_started":1}
I20260928 16:34:51.112392 2595 leader_election.cc:302] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1acf860775654f52a463c0180f8c44b8, b414e4445ad146ffb2452cf8ef09e8f7; no voters: 7a8e7bdef6c440389e1041f44c744b40
I20260928 16:34:51.112406 2619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1acf860775654f52a463c0180f8c44b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" is_pre_election: true
I20260928 16:34:51.112600 2619 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1acf860775654f52a463c0180f8c44b8 in term 1.
I20260928 16:34:51.112680 3139 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 16:34:51.112746 3139 raft_consensus.cc:482] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8e7bdef6c440389e1041f44c744b40)
I20260928 16:34:51.112797 3139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:34:51.112955 2411 leader_election.cc:302] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1acf860775654f52a463c0180f8c44b8, b414e4445ad146ffb2452cf8ef09e8f7; no voters: 7a8e7bdef6c440389e1041f44c744b40
I20260928 16:34:51.113189 3140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 16:34:51.113253 3140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7a8e7bdef6c440389e1041f44c744b40)
I20260928 16:34:51.113291 3140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:34:51.114185 3139 raft_consensus.cc:504] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } }
I20260928 16:34:51.114619 3139 leader_election.cc:288] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [CANDIDATE]: Term 2 election: Requested vote from peers 7a8e7bdef6c440389e1041f44c744b40 (127.27.122.254:41397), 1acf860775654f52a463c0180f8c44b8 (127.27.122.253:44047)
I20260928 16:34:51.114589 3140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } }
I20260928 16:34:51.114903 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1acf860775654f52a463c0180f8c44b8"
I20260928 16:34:51.114960 3140 leader_election.cc:288] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [CANDIDATE]: Term 2 election: Requested vote from peers 7a8e7bdef6c440389e1041f44c744b40 (127.27.122.254:41397), b414e4445ad146ffb2452cf8ef09e8f7 (127.27.122.252:43451)
I20260928 16:34:51.115072 2434 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b414e4445ad146ffb2452cf8ef09e8f7 in current term 2: Already voted for candidate 1acf860775654f52a463c0180f8c44b8 in this term.
I20260928 16:34:51.115089 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.114707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50408)
0928 16:34:51.114851 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:51.114906 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 16:34:51.115262 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.115056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50424)
0928 16:34:51.115149 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:51.115194 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 16:34:51.115559 2619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1acf860775654f52a463c0180f8c44b8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b414e4445ad146ffb2452cf8ef09e8f7"
I20260928 16:34:51.115761 2619 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1acf860775654f52a463c0180f8c44b8 in current term 2: Already voted for candidate b414e4445ad146ffb2452cf8ef09e8f7 in this term.
I20260928 16:34:51.115899 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.115599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50432)
0928 16:34:51.115743 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:51.115841 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 16:34:51.115936 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.115738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50448)
0928 16:34:51.115821 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:51.115869 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 16:34:51.115983 2593 leader_election.cc:334] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7a8e7bdef6c440389e1041f44c744b40 (127.27.122.254:41397): Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
W20260928 16:34:51.116019 2408 leader_election.cc:334] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7a8e7bdef6c440389e1041f44c744b40 (127.27.122.254:41397): Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
I20260928 16:34:51.116041 2593 leader_election.cc:302] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b414e4445ad146ffb2452cf8ef09e8f7; no voters: 1acf860775654f52a463c0180f8c44b8, 7a8e7bdef6c440389e1041f44c744b40
I20260928 16:34:51.116135 2411 leader_election.cc:302] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1acf860775654f52a463c0180f8c44b8; no voters: 7a8e7bdef6c440389e1041f44c744b40, b414e4445ad146ffb2452cf8ef09e8f7
I20260928 16:34:51.116169 3139 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 16:34:51.116288 3140 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 16:34:51.596100 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.595584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.194:52991)
0928 16:34:51.595868 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:51.595972 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":122,"threads_started":1}
I20260928 16:34:51.597126 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.596511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.193:56017)
0928 16:34:51.596884 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:51.597007 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":109,"threads_started":1}
I20260928 16:34:51.600633 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:51.600244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.195:38555)
0928 16:34:51.600457 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:51.600537 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
I20260928 16:34:52.597384 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.596904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.194:45973)
0928 16:34:52.597196 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.597259 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260928 16:34:52.598166 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.597776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.193:44899)
0928 16:34:52.598027 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.598079 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260928 16:34:52.601703 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.601289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.195:42377)
0928 16:34:52.601523 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.601610 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20260928 16:34:52.735308 3155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:52.735452 3155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } }
I20260928 16:34:52.735852 3155 leader_election.cc:288] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7a8e7bdef6c440389e1041f44c744b40 (127.27.122.254:41397), b414e4445ad146ffb2452cf8ef09e8f7 (127.27.122.252:43451)
I20260928 16:34:52.736378 2619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1acf860775654f52a463c0180f8c44b8" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" is_pre_election: true
I20260928 16:34:52.736536 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.736088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50692)
0928 16:34:52.736337 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.736408 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":99,"threads_started":1}
I20260928 16:34:52.736512 2619 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1acf860775654f52a463c0180f8c44b8 in term 2.
I20260928 16:34:52.736974 2411 leader_election.cc:302] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1acf860775654f52a463c0180f8c44b8, b414e4445ad146ffb2452cf8ef09e8f7; no voters:
I20260928 16:34:52.737160 3155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 16:34:52.737231 3155 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:52.737273 3155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:34:52.737504 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.737244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50704)
0928 16:34:52.737395 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.737451 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260928 16:34:52.737766 2408 leader_election.cc:334] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7a8e7bdef6c440389e1041f44c744b40 (127.27.122.254:41397): Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
I20260928 16:34:52.738485 3155 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } }
I20260928 16:34:52.738831 3155 leader_election.cc:288] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [CANDIDATE]: Term 3 election: Requested vote from peers 7a8e7bdef6c440389e1041f44c744b40 (127.27.122.254:41397), b414e4445ad146ffb2452cf8ef09e8f7 (127.27.122.252:43451)
I20260928 16:34:52.739081 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.738901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50720)
0928 16:34:52.738985 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.739036 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 16:34:52.739223 2619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1acf860775654f52a463c0180f8c44b8" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b414e4445ad146ffb2452cf8ef09e8f7"
I20260928 16:34:52.739365 2619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:34:52.739801 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.739579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50732)
0928 16:34:52.739701 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.739749 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 16:34:52.739914 2408 leader_election.cc:334] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7a8e7bdef6c440389e1041f44c744b40 (127.27.122.254:41397): Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
I20260928 16:34:52.740515 2619 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1acf860775654f52a463c0180f8c44b8 in term 3.
I20260928 16:34:52.740813 2411 leader_election.cc:302] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1acf860775654f52a463c0180f8c44b8, b414e4445ad146ffb2452cf8ef09e8f7; no voters: 7a8e7bdef6c440389e1041f44c744b40
I20260928 16:34:52.741009 3155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 16:34:52.741207 3155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [term 3 LEADER]: Becoming Leader. State: Replica: 1acf860775654f52a463c0180f8c44b8, State: Running, Role: LEADER
I20260928 16:34:52.741469 3155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } }
I20260928 16:34:52.742532 3158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1acf860775654f52a463c0180f8c44b8. Latest consensus state: current_term: 3 leader_uuid: "1acf860775654f52a463c0180f8c44b8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } } }
I20260928 16:34:52.742631 3158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:52.742916 3160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:52.745223 3160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:52.745564 3160 catalog_manager.cc:1259] Loaded cluster ID: 90365853bc0a4c12a1cac893085786d0
I20260928 16:34:52.745608 3160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:52.746107 3160 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:52.746449 3160 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: Loaded TSK: 0
I20260928 16:34:52.747175 3160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:34:52.759217 2424 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:34:52.762239 2424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:52.765041 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.764793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50738)
0928 16:34:52.764931 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.764984 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 16:34:52.765172 2619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [term 3 FOLLOWER]: Refusing update from remote peer 1acf860775654f52a463c0180f8c44b8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 16:34:52.765730 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.765478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.0.0.1:50744)
0928 16:34:52.765578 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.765638 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 16:34:52.765651 3158 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 16:34:52.765866 2408 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 -> Peer 7a8e7bdef6c440389e1041f44c744b40 (127.27.122.254:41397): Couldn't send request to peer 7a8e7bdef6c440389e1041f44c744b40. Status: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 16:34:52.767055 3162 sys_catalog.cc:455] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1acf860775654f52a463c0180f8c44b8. Latest consensus state: current_term: 3 leader_uuid: "1acf860775654f52a463c0180f8c44b8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } } }
I20260928 16:34:52.767184 3162 sys_catalog.cc:458] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:34:52.768167 3155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "1acf860775654f52a463c0180f8c44b8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } } }
I20260928 16:34:52.768290 3155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:52.768491 3162 sys_catalog.cc:455] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "1acf860775654f52a463c0180f8c44b8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7a8e7bdef6c440389e1041f44c744b40" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41397 } } peers { permanent_uuid: "1acf860775654f52a463c0180f8c44b8" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 44047 } attrs { promote: false } } peers { permanent_uuid: "b414e4445ad146ffb2452cf8ef09e8f7" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 43451 } attrs { promote: false } } }
I20260928 16:34:52.768599 3162 sys_catalog.cc:458] T 00000000000000000000000000000000 P b414e4445ad146ffb2452cf8ef09e8f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:34:52.778122 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.773996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:38565 (local address 127.0.0.1:57950)
0928 16:34:52.774189 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.774204 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:52.774302 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:52.774731 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:52.774737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:52.774757 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:52.774875 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:52.774905 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.776043 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.776049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:52.776861 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:52.776869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.777746 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.777750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:52.777839 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:52.777864 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:34:52.777909 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 16:34:52.777974 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260928 16:34:52.778795 3167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.774131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.27.122.194:38565)
0928 16:34:52.774458 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:52.774463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:52.774478 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:34:52.774536 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:52.774542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:52.774601 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:52.775052 (+ 451us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:52.775192 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.775942 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.777004 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.777612 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.778492 (+ 880us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:52.778544 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:52.778580 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 16:34:52.778648 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":126,"threads_started":1}
I20260928 16:34:52.779258 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.774492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:45537 (local address 127.0.0.1:53856)
0928 16:34:52.775072 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.775086 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:52.775174 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:52.775615 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:52.775618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:52.775634 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:52.775756 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:52.775761 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.776825 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.776828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:52.778015 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:52.778023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.778825 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.778829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:52.778934 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:52.778964 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:34:52.779031 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 16:34:52.779118 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":7,"spinlock_wait_cycles":9984,"thread_start_us":139,"threads_started":1}
I20260928 16:34:52.779407 3170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.774596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.27.122.195:44807)
0928 16:34:52.775388 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:52.775392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:52.775404 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:34:52.775450 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:52.775453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:52.775507 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:52.775584 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:52.775864 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.776656 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.778155 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.778768 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.778897 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:52.778946 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:52.779159 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 16:34:52.779271 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":91,"threads_started":1}
I20260928 16:34:52.779458 2862 tablet_service.cc:1467] Processing CreateTablet for tablet 74a3beda3c9c47799515f34fe14e899a (DEFAULT_TABLE table=test-table [id=a4183f7b1eca4f27af0cbed098ef56fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:52.779299 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.774485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:44807 (local address 127.0.0.1:43262)
0928 16:34:52.774999 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.775013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:52.775089 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:52.775629 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:52.775632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:52.775652 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:52.775742 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:52.775747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.776798 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.776802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:52.777681 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:52.777691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.778936 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.778939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:52.779035 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:52.779053 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 16:34:52.779141 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:34:52.779197 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":124,"threads_started":1}
I20260928 16:34:52.779800 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a3beda3c9c47799515f34fe14e899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:52.779974 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.774903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.27.122.193:45537)
0928 16:34:52.775342 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:52.775347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:52.775360 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:34:52.775415 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:52.775419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:52.775465 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:52.775887 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:52.776003 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.776736 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.778137 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.778700 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.778835 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:52.778882 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:52.779758 (+ 876us) server_negotiation.cc:299] Negotiation successful
0928 16:34:52.779825 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":83,"threads_started":1}
I20260928 16:34:52.780275 3000 tablet_service.cc:1467] Processing CreateTablet for tablet 74a3beda3c9c47799515f34fe14e899a (DEFAULT_TABLE table=test-table [id=a4183f7b1eca4f27af0cbed098ef56fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:52.780447 2723 tablet_service.cc:1467] Processing CreateTablet for tablet 74a3beda3c9c47799515f34fe14e899a (DEFAULT_TABLE table=test-table [id=a4183f7b1eca4f27af0cbed098ef56fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:52.780516 3000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a3beda3c9c47799515f34fe14e899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:52.780671 2723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a3beda3c9c47799515f34fe14e899a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:52.782532 3172 tablet_bootstrap.cc:492] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c: Bootstrap starting.
I20260928 16:34:52.783239 3173 tablet_bootstrap.cc:492] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296: Bootstrap starting.
I20260928 16:34:52.783480 3172 tablet_bootstrap.cc:654] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:52.784122 3173 tablet_bootstrap.cc:654] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:52.784629 3172 tablet_bootstrap.cc:492] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c: No bootstrap required, opened a new log
I20260928 16:34:52.784737 3172 ts_tablet_manager.cc:1397] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 16:34:52.784766 3174 tablet_bootstrap.cc:492] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6: Bootstrap starting.
I20260928 16:34:52.785223 3172 raft_consensus.cc:348] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } } peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } }
I20260928 16:34:52.785339 3172 raft_consensus.cc:374] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:52.785384 3172 raft_consensus.cc:729] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4419df5c25243b2999c0bec329e149c, State: Initialized, Role: FOLLOWER
I20260928 16:34:52.785561 3174 tablet_bootstrap.cc:654] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:52.785519 3172 consensus_queue.cc:260] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } } peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } }
I20260928 16:34:52.785928 3172 ts_tablet_manager.cc:1428] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:52.786664 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.786397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.194:34963)
0928 16:34:52.786536 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.786588 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 16:34:52.787372 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.787154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.194:42037)
0928 16:34:52.787254 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.787311 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 16:34:52.787887 3173 tablet_bootstrap.cc:492] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296: No bootstrap required, opened a new log
I20260928 16:34:52.787981 3173 ts_tablet_manager.cc:1397] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 16:34:52.788316 2930 heartbeater.cc:503] Master 127.27.122.253:44047 was elected leader, sending a full tablet report...
I20260928 16:34:52.788403 3173 raft_consensus.cc:348] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } } peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } }
I20260928 16:34:52.788527 3173 raft_consensus.cc:374] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:52.788604 3173 raft_consensus.cc:729] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4a75965fcd4f65af7ed425d7d56296, State: Initialized, Role: FOLLOWER
I20260928 16:34:52.788774 3173 consensus_queue.cc:260] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } } peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } }
I20260928 16:34:52.789186 3173 ts_tablet_manager.cc:1428] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:52.789811 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.789546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.193:40151)
0928 16:34:52.789670 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.789737 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 16:34:52.790495 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.790256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.193:36663)
0928 16:34:52.790367 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.790412 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 16:34:52.790771 2792 heartbeater.cc:503] Master 127.27.122.253:44047 was elected leader, sending a full tablet report...
I20260928 16:34:52.791988 3174 tablet_bootstrap.cc:492] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6: No bootstrap required, opened a new log
I20260928 16:34:52.792132 3174 ts_tablet_manager.cc:1397] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260928 16:34:52.792519 3174 raft_consensus.cc:348] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } } peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } }
I20260928 16:34:52.792636 3174 raft_consensus.cc:374] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:52.792680 3174 raft_consensus.cc:729] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5910840c736d43018174009dfc1218f6, State: Initialized, Role: FOLLOWER
I20260928 16:34:52.792811 3174 consensus_queue.cc:260] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } } peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } }
I20260928 16:34:52.793169 3174 ts_tablet_manager.cc:1428] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:52.793708 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.793457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.195:51587)
0928 16:34:52.793585 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.793639 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 16:34:52.794518 3069 heartbeater.cc:503] Master 127.27.122.253:44047 was elected leader, sending a full tablet report...
I20260928 16:34:52.795019 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.794390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.195:34945)
0928 16:34:52.794900 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.794961 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450}
I20260928 16:34:52.799784 3178 raft_consensus.cc:482] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:52.799889 3178 raft_consensus.cc:504] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } } peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } }
I20260928 16:34:52.800223 3178 leader_election.cc:288] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4419df5c25243b2999c0bec329e149c (127.27.122.194:38565), 5910840c736d43018174009dfc1218f6 (127.27.122.195:44807)
I20260928 16:34:52.800570 3020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a3beda3c9c47799515f34fe14e899a" candidate_uuid: "4e4a75965fcd4f65af7ed425d7d56296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5910840c736d43018174009dfc1218f6" is_pre_election: true
I20260928 16:34:52.800686 3020 raft_consensus.cc:2454] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a75965fcd4f65af7ed425d7d56296 in term 0.
I20260928 16:34:52.800800 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a3beda3c9c47799515f34fe14e899a" candidate_uuid: "4e4a75965fcd4f65af7ed425d7d56296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4419df5c25243b2999c0bec329e149c" is_pre_election: true
I20260928 16:34:52.800949 2882 raft_consensus.cc:2454] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4a75965fcd4f65af7ed425d7d56296 in term 0.
I20260928 16:34:52.801047 2676 leader_election.cc:302] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a75965fcd4f65af7ed425d7d56296, 5910840c736d43018174009dfc1218f6; no voters:
I20260928 16:34:52.801235 3178 raft_consensus.cc:2781] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:52.801295 3178 raft_consensus.cc:482] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:52.801349 3178 raft_consensus.cc:3037] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:52.802289 3178 raft_consensus.cc:504] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } } peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } }
I20260928 16:34:52.802635 3178 leader_election.cc:288] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [CANDIDATE]: Term 1 election: Requested vote from peers d4419df5c25243b2999c0bec329e149c (127.27.122.194:38565), 5910840c736d43018174009dfc1218f6 (127.27.122.195:44807)
I20260928 16:34:52.802893 2882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a3beda3c9c47799515f34fe14e899a" candidate_uuid: "4e4a75965fcd4f65af7ed425d7d56296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4419df5c25243b2999c0bec329e149c"
I20260928 16:34:52.802937 3020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a3beda3c9c47799515f34fe14e899a" candidate_uuid: "4e4a75965fcd4f65af7ed425d7d56296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5910840c736d43018174009dfc1218f6"
I20260928 16:34:52.803036 3020 raft_consensus.cc:3037] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:52.803030 2882 raft_consensus.cc:3037] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:52.803994 2882 raft_consensus.cc:2454] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a75965fcd4f65af7ed425d7d56296 in term 1.
I20260928 16:34:52.804049 3020 raft_consensus.cc:2454] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4a75965fcd4f65af7ed425d7d56296 in term 1.
I20260928 16:34:52.804292 2676 leader_election.cc:302] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4a75965fcd4f65af7ed425d7d56296, 5910840c736d43018174009dfc1218f6; no voters:
I20260928 16:34:52.804455 3178 raft_consensus.cc:2781] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:52.804704 3178 raft_consensus.cc:686] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [term 1 LEADER]: Becoming Leader. State: Replica: 4e4a75965fcd4f65af7ed425d7d56296, State: Running, Role: LEADER
I20260928 16:34:52.804857 3178 consensus_queue.cc:237] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } } peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } }
I20260928 16:34:52.805433 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.805152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.193:40195)
0928 16:34:52.805314 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.805362 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 16:34:52.806777 2424 catalog_manager.cc:5495] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4a75965fcd4f65af7ed425d7d56296 (127.27.122.193). New cstate: current_term: 1 leader_uuid: "4e4a75965fcd4f65af7ed425d7d56296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4a75965fcd4f65af7ed425d7d56296" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 45537 } health_report { overall_health: HEALTHY } } }
I20260928 16:34:52.820351 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.817140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.27.122.193:45537)
0928 16:34:52.817259 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:52.817264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:52.817302 (+ 38us) server_negotiation.cc:407] Connection header received
0928 16:34:52.817462 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:52.817465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:52.817515 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:52.817593 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:52.817863 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.818376 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.819334 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:52.819559 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:52.819667 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:52.819931 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:52.820006 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:52.820083 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:52.820178 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 16:34:52.820227 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:34:52.822261 3020 raft_consensus.cc:1261] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6 [term 1 FOLLOWER]: Refusing update from remote peer 4e4a75965fcd4f65af7ed425d7d56296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:52.822628 2882 raft_consensus.cc:1261] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c [term 1 FOLLOWER]: Refusing update from remote peer 4e4a75965fcd4f65af7ed425d7d56296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:52.822705 3181 consensus_queue.cc:1035] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5910840c736d43018174009dfc1218f6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 44807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:52.823060 3181 consensus_queue.cc:1035] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4419df5c25243b2999c0bec329e149c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:52.824219 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.823931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.195:45839)
0928 16:34:52.824071 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.824145 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 16:34:52.824757 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.824467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.194:45155)
0928 16:34:52.824621 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.824685 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 16:34:52.828565 3153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.828124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.193:44447)
0928 16:34:52.828448 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.828500 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256}
I20260928 16:34:52.829174 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.828862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.194:59031)
0928 16:34:52.829041 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.829103 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260928 16:34:52.829264 3154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:52.828949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41397 (local address 127.27.122.195:45337)
0928 16:34:52.829134 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.829182 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:41397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 16:34:52.833468 2422 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:52.833632 2422 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:52.837311 2422 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1acf860775654f52a463c0180f8c44b8: Sending DeleteTablet for 3 replicas of tablet 74a3beda3c9c47799515f34fe14e899a
I20260928 16:34:52.837894 2862 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a3beda3c9c47799515f34fe14e899a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:52 UTC) from {username='oryx', principal='oryx/127.27.122.253@KRBTEST.COM'} at 127.0.0.1:57950
I20260928 16:34:52.838039 2723 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a3beda3c9c47799515f34fe14e899a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:52 UTC) from {username='oryx', principal='oryx/127.27.122.253@KRBTEST.COM'} at 127.0.0.1:53856
I20260928 16:34:52.838052 3000 tablet_service.cc:1514] Processing DeleteTablet for tablet 74a3beda3c9c47799515f34fe14e899a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:52 UTC) from {username='oryx', principal='oryx/127.27.122.253@KRBTEST.COM'} at 127.0.0.1:43262
I20260928 16:34:52.838356 3196 tablet_replica.cc:330] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296: stopping tablet replica
I20260928 16:34:52.838392 3197 tablet_replica.cc:330] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6: stopping tablet replica
I20260928 16:34:52.838507 3195 tablet_replica.cc:330] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c: stopping tablet replica
I20260928 16:34:52.838536 3196 raft_consensus.cc:2229] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:52.838567 3197 raft_consensus.cc:2229] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:52.838681 3195 raft_consensus.cc:2229] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:52.838742 3197 raft_consensus.cc:2258] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:52.838861 3195 raft_consensus.cc:2258] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:52.838878 3196 raft_consensus.cc:2258] T 74a3beda3c9c47799515f34fe14e899a P 4e4a75965fcd4f65af7ed425d7d56296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:52.839265 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 2569
I20260928 16:34:52.840288 3197 ts_tablet_manager.cc:1905] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:52.842092 3195 ts_tablet_manager.cc:1905] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:52.842969 3197 ts_tablet_manager.cc:1918] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:52.843065 3197 log.cc:1196] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/74a3beda3c9c47799515f34fe14e899a
I20260928 16:34:52.843360 3197 ts_tablet_manager.cc:1939] T 74a3beda3c9c47799515f34fe14e899a P 5910840c736d43018174009dfc1218f6: Deleting consensus metadata
I20260928 16:34:52.844107 2409 catalog_manager.cc:4841] TS 5910840c736d43018174009dfc1218f6 (127.27.122.195:44807): tablet 74a3beda3c9c47799515f34fe14e899a (table test-table [id=a4183f7b1eca4f27af0cbed098ef56fa]) successfully deleted
I20260928 16:34:52.845360 3195 ts_tablet_manager.cc:1918] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:52.845446 3195 log.cc:1196] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/74a3beda3c9c47799515f34fe14e899a
I20260928 16:34:52.845750 3195 ts_tablet_manager.cc:1939] T 74a3beda3c9c47799515f34fe14e899a P d4419df5c25243b2999c0bec329e149c: Deleting consensus metadata
I20260928 16:34:52.846464 2411 catalog_manager.cc:4841] TS d4419df5c25243b2999c0bec329e149c (127.27.122.194:38565): tablet 74a3beda3c9c47799515f34fe14e899a (table test-table [id=a4183f7b1eca4f27af0cbed098ef56fa]) successfully deleted
W20260928 16:34:52.849200 3168 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:34:52.848866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:45537 (local address 127.0.0.1:53868)
0928 16:34:52.849018 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:52.849064 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:45537: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 16:34:52.849203 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 2798
W20260928 16:34:52.853053 2408 catalog_manager.cc:4584] TS 4e4a75965fcd4f65af7ed425d7d56296 (127.27.122.193:45537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74a3beda3c9c47799515f34fe14e899a: Network error: Client connection negotiation failed: client connection to 127.27.122.193:45537: connect: Connection reset by peer (error 104)
I20260928 16:34:52.855664 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 2935
I20260928 16:34:52.862082 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 2280
I20260928 16:34:52.868228 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 2389
2026-09-28T16:34:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6854 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:52 dist-test-slave-g9sh krb5kdc[3200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:52 dist-test-slave-g9sh krb5kdc[3200](info): set up 2 sockets
Sep 28 16:34:52 dist-test-slave-g9sh krb5kdc[3200](info): commencing operation
krb5kdc: starting...
W20260928 16:34:53.916647 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:54 dist-test-slave-g9sh krb5kdc[3200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.254.keytab.
Entry for principal oryx/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.254.keytab.
I20260928 16:34:54.036753 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:45455
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.254.keytab
--principal=oryx/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:45455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:54.147526 3213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:54.147814 3213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:54.147886 3213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:54.151916 3213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:54.152004 3213 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 16:34:54.152034 3213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:54.152078 3213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:54.152122 3213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:54.152158 3213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:54.157786 3213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:45455
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.254.keytab
--principal=oryx/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:45455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:54.158811 3213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:54.159832 3213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:54.168143 3221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:54.168205 3213 server_base.cc:1034] running on GCE node
W20260928 16:34:54.168149 3218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:54.168149 3219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:54.169075 3213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:54.170104 3213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:54.171270 3213 hybrid_clock.cc:648] HybridClock initialized: now 1790613294171236 us; error 44 us; skew 500 ppm
Sep 28 16:34:54 dist-test-slave-g9sh krb5kdc[3200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613294, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:54.178911 3213 init.cc:377] Logged in from keytab as oryx/127.27.122.254@KRBTEST.COM (short username oryx)
I20260928 16:34:54.179831 3213 webserver.cc:460] Webserver started at http://127.27.122.254:39785/ using document root <none> and password file <none>
I20260928 16:34:54.180410 3213 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:54.180503 3213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:54.180716 3213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:54.182881 3213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "04dae37594d24bbc9ba551d068f4b794"
format_stamp: "Formatted at 2026-09-28 16:34:54 on dist-test-slave-g9sh"
server_key: "15da58631be0a59c8776c2a8b28f0bfe"
server_key_iv: "8c8b992ef677d3d7c3cde32e00af8b1a"
server_key_version: "encryptionkey@0"
I20260928 16:34:54.183478 3213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "04dae37594d24bbc9ba551d068f4b794"
format_stamp: "Formatted at 2026-09-28 16:34:54 on dist-test-slave-g9sh"
server_key: "15da58631be0a59c8776c2a8b28f0bfe"
server_key_iv: "8c8b992ef677d3d7c3cde32e00af8b1a"
server_key_version: "encryptionkey@0"
I20260928 16:34:54.187299 3213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 16:34:54.189806 3228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:54.191020 3213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:54.191206 3213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "04dae37594d24bbc9ba551d068f4b794"
format_stamp: "Formatted at 2026-09-28 16:34:54 on dist-test-slave-g9sh"
server_key: "15da58631be0a59c8776c2a8b28f0bfe"
server_key_iv: "8c8b992ef677d3d7c3cde32e00af8b1a"
server_key_version: "encryptionkey@0"
I20260928 16:34:54.191327 3213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:54.208091 3213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:54.210992 3213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:54.211210 3213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:54.219492 3213 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:45455
I20260928 16:34:54.219515 3290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:45455 every 8 connection(s)
I20260928 16:34:54.220445 3213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:54.222995 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 3213
I20260928 16:34:54.223126 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:54.223254 3291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:54.223464 28139 external_mini_cluster.cc:1428] Setting key 3ff0724931ca8fb6ad5ce88298a521d4
I20260928 16:34:54.230585 3291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794: Bootstrap starting.
Sep 28 16:34:54 dist-test-slave-g9sh krb5kdc[3200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
I20260928 16:34:54.233144 3291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:54.234053 3291 log.cc:824] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:54.236313 3291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794: No bootstrap required, opened a new log
I20260928 16:34:54.239048 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.224877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.27.122.254:45455)
0928 16:34:54.225370 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:54.225387 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:54.225431 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:34:54.226098 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:54.226120 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:54.226533 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:54.226895 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:54.227551 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.229070 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.229910 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.230576 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.230742 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.233451 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:54.233458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:54.233474 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:54.233505 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:54.235893 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.236352 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.236360 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.236367 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.236499 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.236806 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.236809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.236810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.237337 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:54.237537 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:54.237822 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 16:34:54.238118 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":121,"threads_started":1}
I20260928 16:34:54.240283 3291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04dae37594d24bbc9ba551d068f4b794" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 45455 } }
I20260928 16:34:54.240536 3291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:54.240603 3291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04dae37594d24bbc9ba551d068f4b794, State: Initialized, Role: FOLLOWER
I20260928 16:34:54.241073 3291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04dae37594d24bbc9ba551d068f4b794" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 45455 } }
I20260928 16:34:54.241243 3291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:54.241340 3291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:54.241448 3291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:54.242313 3291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04dae37594d24bbc9ba551d068f4b794" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 45455 } }
I20260928 16:34:54.242656 3291 leader_election.cc:302] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04dae37594d24bbc9ba551d068f4b794; no voters:
I20260928 16:34:54.242944 3291 leader_election.cc:288] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:54.243069 3296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:54.243315 3296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [term 1 LEADER]: Becoming Leader. State: Replica: 04dae37594d24bbc9ba551d068f4b794, State: Running, Role: LEADER
I20260928 16:34:54.243682 3296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04dae37594d24bbc9ba551d068f4b794" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 45455 } }
I20260928 16:34:54.244153 3291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:54.245517 3297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04dae37594d24bbc9ba551d068f4b794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04dae37594d24bbc9ba551d068f4b794" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 45455 } } }
I20260928 16:34:54.245498 3298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04dae37594d24bbc9ba551d068f4b794. Latest consensus state: current_term: 1 leader_uuid: "04dae37594d24bbc9ba551d068f4b794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04dae37594d24bbc9ba551d068f4b794" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 45455 } } }
I20260928 16:34:54.245671 3298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:54.245671 3297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794 [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:54.246052 3306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:54.249486 3306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:54.255007 3306 catalog_manager.cc:1347] Generated new cluster ID: 40602fc52f7a459789ba6746da11ba44
I20260928 16:34:54.255085 3306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:54.273131 3306 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:54.274112 3306 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:54.298341 3306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794: Generated new TSK 0
I20260928 16:34:54.299099 3306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab.
Entry for principal oryx/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab.
I20260928 16:34:54.330340 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:45455
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab
--principal=oryx/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:54.440346 3317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:54.440594 3317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:54.440637 3317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:54.444634 3317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:54.444697 3317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:54.444736 3317 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:34:54.444767 3317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:54.450196 3317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab
--principal=oryx/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.122.254:45455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:54.451130 3317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:54.452064 3317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:54.461431 3323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:54.461498 3322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:54.461493 3317 server_base.cc:1034] running on GCE node
W20260928 16:34:54.461525 3325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:54.462141 3317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:54.462800 3317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:54.463982 3317 hybrid_clock.cc:648] HybridClock initialized: now 1790613294463947 us; error 44 us; skew 500 ppm
Sep 28 16:34:54 dist-test-slave-g9sh krb5kdc[3200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613294, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:54.466711 3317 init.cc:377] Logged in from keytab as oryx/127.27.122.193@KRBTEST.COM (short username oryx)
I20260928 16:34:54.467485 3317 webserver.cc:460] Webserver started at http://127.27.122.193:43479/ using document root <none> and password file <none>
I20260928 16:34:54.468168 3317 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:54.468271 3317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:54.468528 3317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:54.470679 3317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "18222ddaedc74c41b6f47face5c2b261"
format_stamp: "Formatted at 2026-09-28 16:34:54 on dist-test-slave-g9sh"
server_key: "388d4da01af218aa78dd8fd14e326ead"
server_key_iv: "5567ae6005e2d4e039f8a6dddc986412"
server_key_version: "encryptionkey@0"
I20260928 16:34:54.471259 3317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "18222ddaedc74c41b6f47face5c2b261"
format_stamp: "Formatted at 2026-09-28 16:34:54 on dist-test-slave-g9sh"
server_key: "388d4da01af218aa78dd8fd14e326ead"
server_key_iv: "5567ae6005e2d4e039f8a6dddc986412"
server_key_version: "encryptionkey@0"
I20260928 16:34:54.474960 3317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:34:54.477447 3332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:54.478605 3317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:54.478740 3317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "18222ddaedc74c41b6f47face5c2b261"
format_stamp: "Formatted at 2026-09-28 16:34:54 on dist-test-slave-g9sh"
server_key: "388d4da01af218aa78dd8fd14e326ead"
server_key_iv: "5567ae6005e2d4e039f8a6dddc986412"
server_key_version: "encryptionkey@0"
I20260928 16:34:54.478852 3317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:54.488603 3317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:54.491392 3317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:54.491588 3317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:54.493047 3317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:54.493134 3317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:54.493207 3317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:54.493242 3317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:34:54 dist-test-slave-g9sh krb5kdc[3200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613294, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
I20260928 16:34:54.504472 3317 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:40065
I20260928 16:34:54.504776 3448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:40065 every 8 connection(s)
I20260928 16:34:54.505581 3317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:54.506315 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 3317
I20260928 16:34:54.506456 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:34:54.506784 28139 external_mini_cluster.cc:1428] Setting key 12a7678a30d8328052f7a5fb64184487
I20260928 16:34:54.508440 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.493888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:46125 (local address 127.27.122.254:45455)
0928 16:34:54.494042 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:54.494047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:54.494890 (+ 843us) server_negotiation.cc:407] Connection header received
0928 16:34:54.496061 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:54.496065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:54.496129 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:54.496193 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:54.497961 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.498526 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.499693 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.499935 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.500062 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.503614 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:54.503617 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:54.503619 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:54.503644 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:54.505247 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.505858 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.505862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.505863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.505944 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.507229 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.507233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.507235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.507451 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:54.507532 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:54.508142 (+ 610us) server_negotiation.cc:299] Negotiation successful
0928 16:34:54.508268 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:34:54.509291 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.494285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:45455 (local address 127.27.122.193:46125)
0928 16:34:54.494732 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:54.494779 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:54.495741 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:54.496688 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:54.496697 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:54.497251 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:54.497803 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:54.497820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.498663 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.498667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:54.499560 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:54.499569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.500109 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.500113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:54.500290 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.500999 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:54.501018 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:54.503434 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:54.505362 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:54.505367 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:54.505381 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:54.505746 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:54.506085 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:54.506090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:54.506092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:54.506263 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:54.507542 (+ 1279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:54.507548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:54.507857 (+ 309us) client_negotiation.cc:769] Sending connection context
0928 16:34:54.508093 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 16:34:54.508408 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":134,"threads_started":1}
I20260928 16:34:54.510799 3449 heartbeater.cc:347] Connected to a master server at 127.27.122.254:45455
I20260928 16:34:54.510936 3449 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:54.511507 3449 heartbeater.cc:511] Master 127.27.122.254:45455 requested a full tablet report, sending...
I20260928 16:34:54.513319 3244 ts_manager.cc:194] Registered new tserver with Master: 18222ddaedc74c41b6f47face5c2b261 (127.27.122.193:40065)
I20260928 16:34:54.515045 3244 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:46125
WARNING: no policy specified for oryx/127.27.122.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.122.194@KRBTEST.COM" created.
I20260928 16:34:54.524430 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.513129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:52349 (local address 127.27.122.254:45455)
0928 16:34:54.513251 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:54.513255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:54.517500 (+ 4245us) server_negotiation.cc:407] Connection header received
0928 16:34:54.517653 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:54.517657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:54.517706 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:54.517789 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:54.518208 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.518711 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.519547 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.519865 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.519971 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.521220 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:54.521226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:54.521231 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:54.521265 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:54.522714 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.523209 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.523212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.523213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.523282 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.523608 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.523612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.523616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.523792 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:54.523868 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:54.524218 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 16:34:54.524315 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:34:54.524616 3456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.513111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:45455 (local address 127.27.122.193:52349)
0928 16:34:54.517413 (+ 4302us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:54.517425 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:54.517548 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:54.517806 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:54.517809 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:54.517997 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:54.518093 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:54.518098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.518839 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.518847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:54.519417 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:54.519425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.520008 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.520011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:54.520092 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.520644 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:54.520656 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:54.521045 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:54.522828 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:54.522834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:54.522839 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:54.523107 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:54.523373 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:54.523379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:54.523384 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:54.523491 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:54.523895 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:54.523898 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:54.524008 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 16:34:54.524130 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 16:34:54.524305 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4229,"thread_start_us":91,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab.
Entry for principal oryx/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab.
I20260928 16:34:54.536521 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:45455
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab
--principal=oryx/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:54.648030 3458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:54.648329 3458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:54.648404 3458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:54.652813 3458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:54.652868 3458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:54.652967 3458 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:34:54.653030 3458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:54.658181 3458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab
--principal=oryx/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.122.254:45455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:54.659204 3458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:54.660213 3458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:54.669721 3463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:54.669770 3466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:54.669749 3464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:54.669970 3458 server_base.cc:1034] running on GCE node
I20260928 16:34:54.670521 3458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:54.671175 3458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:54.672361 3458 hybrid_clock.cc:648] HybridClock initialized: now 1790613294672325 us; error 49 us; skew 500 ppm
Sep 28 16:34:54 dist-test-slave-g9sh krb5kdc[3200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613294, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:54.675222 3458 init.cc:377] Logged in from keytab as oryx/127.27.122.194@KRBTEST.COM (short username oryx)
I20260928 16:34:54.676106 3458 webserver.cc:460] Webserver started at http://127.27.122.194:35319/ using document root <none> and password file <none>
I20260928 16:34:54.676680 3458 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:54.676775 3458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:54.677062 3458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:54.679056 3458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "4ef0a417149a4e8cb260476daacd441a"
format_stamp: "Formatted at 2026-09-28 16:34:54 on dist-test-slave-g9sh"
server_key: "fde5bda66ece53ff1881d33b8ac8481f"
server_key_iv: "46dbfba0ae4e597cac6bfefd24f1ac07"
server_key_version: "encryptionkey@0"
I20260928 16:34:54.679646 3458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "4ef0a417149a4e8cb260476daacd441a"
format_stamp: "Formatted at 2026-09-28 16:34:54 on dist-test-slave-g9sh"
server_key: "fde5bda66ece53ff1881d33b8ac8481f"
server_key_iv: "46dbfba0ae4e597cac6bfefd24f1ac07"
server_key_version: "encryptionkey@0"
I20260928 16:34:54.683444 3458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 16:34:54.685935 3473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:54.687179 3458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:54.687321 3458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "4ef0a417149a4e8cb260476daacd441a"
format_stamp: "Formatted at 2026-09-28 16:34:54 on dist-test-slave-g9sh"
server_key: "fde5bda66ece53ff1881d33b8ac8481f"
server_key_iv: "46dbfba0ae4e597cac6bfefd24f1ac07"
server_key_version: "encryptionkey@0"
I20260928 16:34:54.687433 3458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:54.702325 3458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:54.705274 3458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:54.705472 3458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:54.706910 3458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:54.706983 3458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:54.707057 3458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:54.707115 3458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:54.717967 3458 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:43365
Sep 28 16:34:54 dist-test-slave-g9sh krb5kdc[3200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613294, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
I20260928 16:34:54.718058 3589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:43365 every 8 connection(s)
I20260928 16:34:54.719007 3458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:54.722359 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.707632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:49075 (local address 127.27.122.254:45455)
0928 16:34:54.707767 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:54.707771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:54.709447 (+ 1676us) server_negotiation.cc:407] Connection header received
0928 16:34:54.710814 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:54.710818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:54.710880 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:54.710974 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:54.712728 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.713465 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.715302 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.715548 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.715636 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.718790 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:54.718794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:54.718796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:54.718824 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:54.720508 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.721024 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.721028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.721029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.721117 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.721460 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.721463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.721464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.721636 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:54.721713 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:54.722139 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 16:34:54.722238 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:34:54.722715 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 3458
I20260928 16:34:54.722862 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:34:54.723181 28139 external_mini_cluster.cc:1428] Setting key d7cf978c44e479d532abf911a0e26235
I20260928 16:34:54.723282 3482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.708718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:45455 (local address 127.27.122.194:49075)
0928 16:34:54.709288 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:54.709330 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:54.710588 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:54.711209 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:54.711223 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:54.711796 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:54.712536 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:54.712560 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.713611 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.713616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:54.715159 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:54.715173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.715718 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.715725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:54.715961 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.716674 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:54.716699 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:54.718586 (+ 1887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:54.720614 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:54.720618 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:54.720628 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:54.720873 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:54.721208 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:54.721210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:54.721211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:54.721342 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:54.721724 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:54.721729 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:54.721935 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 16:34:54.722076 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 16:34:54.722313 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":162,"threads_started":1}
I20260928 16:34:54.724666 3590 heartbeater.cc:347] Connected to a master server at 127.27.122.254:45455
I20260928 16:34:54.724810 3590 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:54.725390 3590 heartbeater.cc:511] Master 127.27.122.254:45455 requested a full tablet report, sending...
I20260928 16:34:54.726547 3245 ts_manager.cc:194] Registered new tserver with Master: 4ef0a417149a4e8cb260476daacd441a (127.27.122.194:43365)
I20260928 16:34:54.727350 3245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:49075
I20260928 16:34:54.734550 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.726839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:45455 (local address 127.27.122.194:58379)
0928 16:34:54.727138 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:54.727152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:54.727242 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:54.727513 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:54.727517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:54.727746 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:54.727903 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:54.727913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.728934 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.728940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:54.729561 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:54.729570 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.730195 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.730198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:54.730265 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.730781 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:54.730799 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:54.731101 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:54.733113 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:54.733116 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:54.733117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:54.733302 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:54.733607 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:54.733609 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:54.733611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:54.733682 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:54.734181 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:54.734182 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:54.734252 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 16:34:54.734329 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:34:54.734429 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":109,"threads_started":1}
I20260928 16:34:54.734717 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.726908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:58379 (local address 127.27.122.254:45455)
0928 16:34:54.727027 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:54.727031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:54.727204 (+ 173us) server_negotiation.cc:407] Connection header received
0928 16:34:54.727319 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:54.727322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:54.727357 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:54.727425 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:54.728016 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.728766 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.729683 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.730076 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.730192 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.731235 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:54.731240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:54.731243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:54.731271 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:54.732999 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.733416 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.733420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.733422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.733514 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.733775 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.733779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.733781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.734076 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:54.734193 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:54.734432 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 16:34:54.734553 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for oryx/127.27.122.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.122.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab.
Entry for principal oryx/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab.
I20260928 16:34:54.753444 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:45455
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab
--principal=oryx/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:54.866397 3599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:54.866637 3599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:54.866684 3599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:54.870731 3599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:54.870795 3599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:54.870837 3599 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 16:34:54.870878 3599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:54.876042 3599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab
--principal=oryx/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.122.254:45455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:54.877144 3599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:54.878129 3599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:34:54.887907 3599 server_base.cc:1034] running on GCE node
W20260928 16:34:54.887924 3605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:54.887841 3607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:54.887841 3604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:54.888505 3599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:54.889191 3599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:54.890348 3599 hybrid_clock.cc:648] HybridClock initialized: now 1790613294890324 us; error 61 us; skew 500 ppm
Sep 28 16:34:54 dist-test-slave-g9sh krb5kdc[3200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613294, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:54.893150 3599 init.cc:377] Logged in from keytab as oryx/127.27.122.195@KRBTEST.COM (short username oryx)
I20260928 16:34:54.893983 3599 webserver.cc:460] Webserver started at http://127.27.122.195:36665/ using document root <none> and password file <none>
I20260928 16:34:54.894541 3599 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:54.894603 3599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:54.894817 3599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:54.897008 3599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c"
format_stamp: "Formatted at 2026-09-28 16:34:54 on dist-test-slave-g9sh"
server_key: "35429f3e621a52a3838f6c67dba7178b"
server_key_iv: "1eb787894ee583ee488b8608182bcabb"
server_key_version: "encryptionkey@0"
I20260928 16:34:54.897529 3599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c"
format_stamp: "Formatted at 2026-09-28 16:34:54 on dist-test-slave-g9sh"
server_key: "35429f3e621a52a3838f6c67dba7178b"
server_key_iv: "1eb787894ee583ee488b8608182bcabb"
server_key_version: "encryptionkey@0"
I20260928 16:34:54.901262 3599 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:34:54.903568 3614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:54.904765 3599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:54.904860 3599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c"
format_stamp: "Formatted at 2026-09-28 16:34:54 on dist-test-slave-g9sh"
server_key: "35429f3e621a52a3838f6c67dba7178b"
server_key_iv: "1eb787894ee583ee488b8608182bcabb"
server_key_version: "encryptionkey@0"
I20260928 16:34:54.905007 3599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:54.926364 3599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:54.929210 3599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:54.929375 3599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:54.930708 3599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:54.930759 3599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:54.930799 3599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:54.930857 3599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 16:34:54 dist-test-slave-g9sh krb5kdc[3200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613294, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
I20260928 16:34:54.941870 3599 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:41173
I20260928 16:34:54.941931 3730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:41173 every 8 connection(s)
I20260928 16:34:54.943002 3599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:34:54.945188 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.931608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:33469 (local address 127.27.122.254:45455)
0928 16:34:54.931739 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:54.931743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:54.932794 (+ 1051us) server_negotiation.cc:407] Connection header received
0928 16:34:54.934048 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:54.934052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:54.934131 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:54.934239 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:54.935789 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.936696 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.938093 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.938335 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.938461 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.941281 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:54.941284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:54.941287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:54.941315 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:54.943023 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.943572 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.943576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.943578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.943672 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.944037 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.944041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.944043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.944232 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:54.944343 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:54.944858 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 16:34:54.945021 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:34:54.945878 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.932141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:45455 (local address 127.27.122.195:33469)
0928 16:34:54.932638 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:54.932678 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:54.933733 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:54.934496 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:54.934507 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:54.934923 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:54.935612 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:54.935630 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.936849 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.936853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:54.937965 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:54.937976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.938506 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.938510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:54.938686 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.939276 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:54.939292 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:54.941113 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:54.943148 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:54.943155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:54.943168 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:54.943419 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:54.943778 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:54.943781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:54.943783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:54.943927 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:54.944340 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:54.944346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:54.944598 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 16:34:54.944776 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 16:34:54.945104 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":142,"threads_started":1}
I20260928 16:34:54.947104 3731 heartbeater.cc:347] Connected to a master server at 127.27.122.254:45455
I20260928 16:34:54.947263 3731 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:54.947820 3731 heartbeater.cc:511] Master 127.27.122.254:45455 requested a full tablet report, sending...
I20260928 16:34:54.948891 3243 ts_manager.cc:194] Registered new tserver with Master: 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195:41173)
I20260928 16:34:54.949659 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 3599
I20260928 16:34:54.949661 3243 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:33469
I20260928 16:34:54.949812 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:34:54.950192 28139 external_mini_cluster.cc:1428] Setting key 1f68b51448307889a9a5464df18d3da1
I20260928 16:34:54.952234 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:34:54.957747 3737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.949216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:45455 (local address 127.27.122.195:55735)
0928 16:34:54.950100 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:54.950115 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:54.950212 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:54.950487 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:54.950490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:54.950737 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:54.950981 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:54.950989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.951979 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.951983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:54.952620 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:54.952629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.953169 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.953173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:54.953281 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.953816 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:54.953828 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:54.954117 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:54.955969 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:54.955976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:54.955981 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:54.956261 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:54.956608 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:54.956614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:54.956619 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:54.956730 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:54.957185 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:54.957188 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:54.957301 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 16:34:54.957414 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 16:34:54.957569 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":103,"threads_started":1}
I20260928 16:34:54.957995 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.949289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:55735 (local address 127.27.122.254:45455)
0928 16:34:54.949398 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:54.949402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:54.950196 (+ 794us) server_negotiation.cc:407] Connection header received
0928 16:34:54.950338 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:54.950346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:54.950386 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:54.950472 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:54.951119 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.951826 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.952741 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.953051 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.953167 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.954253 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:54.954257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:54.954259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:54.954282 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:54.955839 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.956380 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.956383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.956384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.956466 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.956834 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.956836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.956837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.957079 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:54.957169 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:54.957499 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 16:34:54.957650 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 16:34:54 dist-test-slave-g9sh krb5kdc[3200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 16:34:54 dist-test-slave-g9sh krb5kdc[3200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:34:54 dist-test-slave-g9sh krb5kdc[3200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:34:54.976717 3294 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:34:54.970673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.27.122.254:45455)
0928 16:34:54.970816 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:54.970825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:54.970836 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:34:54.970866 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:54.970868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:54.970909 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:54.970995 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:54.971683 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.972192 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.973127 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.973344 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.973473 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.976547 (+ 3074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59260: BlockingRecv error: recv got EOF from 127.0.0.1:59260 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 16:34:54 dist-test-slave-g9sh krb5kdc[3200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
I20260928 16:34:54.986088 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:54.978097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59262 (local address 127.27.122.254:45455)
0928 16:34:54.978233 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:54.978237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:54.978362 (+ 125us) server_negotiation.cc:407] Connection header received
0928 16:34:54.978495 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:54.978502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:54.978538 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:54.978622 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:54.979224 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.979702 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.980495 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:54.980713 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:54.980818 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:54.983084 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:54.983088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:54.983091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:54.983124 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:54.984398 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.984843 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.984847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.984848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.984974 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:54.985253 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:54.985257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:54.985259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:54.985433 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:54.985579 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:54.985810 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 16:34:54.985919 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:34:54.989233 3243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:34:54.992524 3243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:55.007544 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.001820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.27.122.195:41173)
0928 16:34:55.002587 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.002600 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.002826 (+ 226us) server_negotiation.cc:407] Connection header received
0928 16:34:55.002969 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.002974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.003131 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.003288 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:55.003546 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.004260 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.005829 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.006593 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.006732 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.006879 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.007283 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.007394 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":121,"threads_started":1}
I20260928 16:34:55.007534 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.001386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:41173 (local address 127.0.0.1:47606)
0928 16:34:55.002724 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:55.002735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:55.002847 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:55.003273 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:55.003276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:55.003294 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:55.003424 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.003431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.004384 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.004388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.005704 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.005713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.006952 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.006959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.007065 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.007111 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 16:34:55.007232 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 16:34:55.007384 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1279,"mutex_wait_us":8,"thread_start_us":99,"threads_started":1}
I20260928 16:34:55.007615 3755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.001159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40936 (local address 127.27.122.194:43365)
0928 16:34:55.002366 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.002379 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.002406 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:34:55.002488 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.002493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.002711 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.003217 (+ 506us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:55.003366 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.004509 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.006108 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.006905 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.007037 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.007152 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.007383 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.007465 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1100,"thread_start_us":124,"threads_started":1}
I20260928 16:34:55.007603 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.001927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.27.122.193:40065)
0928 16:34:55.002379 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.002394 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.002412 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:34:55.002487 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.002493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.002711 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.002875 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:55.003283 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.004523 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.006146 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.006670 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.006833 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.006969 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.007352 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.007452 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":92,"threads_started":1}
I20260928 16:34:55.007977 3754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.001117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40065 (local address 127.0.0.1:41496)
0928 16:34:55.001873 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:55.001928 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:55.002065 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:55.003031 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:55.003034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:55.003047 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:55.003158 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.003165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.004920 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.004923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.006007 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.006016 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.006800 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.006804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.007046 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.007149 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 16:34:55.007225 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 16:34:55.007861 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":570,"spinlock_wait_cycles":7168,"thread_start_us":84,"threads_started":1}
I20260928 16:34:55.007977 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.001027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:43365 (local address 127.0.0.1:40936)
0928 16:34:55.001873 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:55.001929 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:55.002065 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:55.002897 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:55.002903 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:55.002926 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:55.003053 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.003059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.004684 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.004686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.005972 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.005984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.007050 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.007053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.007209 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.007236 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:34:55.007298 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 16:34:55.007824 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":429,"spinlock_wait_cycles":3712,"thread_start_us":80,"threads_started":1}
I20260928 16:34:55.010922 3383 tablet_service.cc:1467] Processing CreateTablet for tablet 86ca3ed4f8a94ba090455eaae8eb32f2 (DEFAULT_TABLE table=test-table [id=5e220669c1f34a1ab54c41a16a0797b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:55.010980 3665 tablet_service.cc:1467] Processing CreateTablet for tablet 86ca3ed4f8a94ba090455eaae8eb32f2 (DEFAULT_TABLE table=test-table [id=5e220669c1f34a1ab54c41a16a0797b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:55.010980 3524 tablet_service.cc:1467] Processing CreateTablet for tablet 86ca3ed4f8a94ba090455eaae8eb32f2 (DEFAULT_TABLE table=test-table [id=5e220669c1f34a1ab54c41a16a0797b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:55.011998 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ca3ed4f8a94ba090455eaae8eb32f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.012010 3665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ca3ed4f8a94ba090455eaae8eb32f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.012008 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ca3ed4f8a94ba090455eaae8eb32f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.018038 3759 tablet_bootstrap.cc:492] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261: Bootstrap starting.
I20260928 16:34:55.019651 3760 tablet_bootstrap.cc:492] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a: Bootstrap starting.
I20260928 16:34:55.020522 3761 tablet_bootstrap.cc:492] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Bootstrap starting.
I20260928 16:34:55.020823 3759 tablet_bootstrap.cc:654] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.021714 3760 tablet_bootstrap.cc:654] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.021747 3759 log.cc:824] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:55.022343 3761 tablet_bootstrap.cc:654] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.022540 3760 log.cc:824] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:55.023166 3761 log.cc:824] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:55.024297 3759 tablet_bootstrap.cc:492] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261: No bootstrap required, opened a new log
I20260928 16:34:55.024489 3759 ts_tablet_manager.cc:1397] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 16:34:55.024530 3760 tablet_bootstrap.cc:492] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a: No bootstrap required, opened a new log
I20260928 16:34:55.024695 3760 ts_tablet_manager.cc:1397] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 16:34:55.025151 3761 tablet_bootstrap.cc:492] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c: No bootstrap required, opened a new log
I20260928 16:34:55.025321 3761 ts_tablet_manager.cc:1397] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:34:55.027558 3759 raft_consensus.cc:348] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.027802 3759 raft_consensus.cc:374] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.027856 3759 raft_consensus.cc:729] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18222ddaedc74c41b6f47face5c2b261, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.028496 3759 consensus_queue.cc:260] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.028800 3760 raft_consensus.cc:348] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.029070 3760 raft_consensus.cc:374] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.029134 3760 raft_consensus.cc:729] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef0a417149a4e8cb260476daacd441a, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.029026 3761 raft_consensus.cc:348] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.029250 3761 raft_consensus.cc:374] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.029292 3761 raft_consensus.cc:729] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e6c90c6b1fe4f209af81e4d18b9a38c, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.029531 3759 ts_tablet_manager.cc:1428] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 16:34:55.029584 3760 consensus_queue.cc:260] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.029666 3761 consensus_queue.cc:260] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.029891 3449 heartbeater.cc:503] Master 127.27.122.254:45455 was elected leader, sending a full tablet report...
I20260928 16:34:55.030201 3590 heartbeater.cc:503] Master 127.27.122.254:45455 was elected leader, sending a full tablet report...
I20260928 16:34:55.030529 3761 ts_tablet_manager.cc:1428] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:34:55.030529 3760 ts_tablet_manager.cc:1428] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:34:55.030900 3731 heartbeater.cc:503] Master 127.27.122.254:45455 was elected leader, sending a full tablet report...
I20260928 16:34:55.169356 3767 raft_consensus.cc:482] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:55.169560 3767 raft_consensus.cc:504] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.170866 3767 leader_election.cc:288] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ef0a417149a4e8cb260476daacd441a (127.27.122.194:43365), 18222ddaedc74c41b6f47face5c2b261 (127.27.122.193:40065)
I20260928 16:34:55.175007 3755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.171286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:33303 (local address 127.27.122.194:43365)
0928 16:34:55.171403 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.171407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.171835 (+ 428us) server_negotiation.cc:407] Connection header received
0928 16:34:55.171897 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.171899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.171940 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.172031 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:55.172331 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.172841 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.174006 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.174413 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.174548 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.174616 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.174796 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.174872 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:34:55.175046 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.171261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:36117 (local address 127.27.122.193:40065)
0928 16:34:55.171405 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.171411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.171432 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:34:55.171532 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.171535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.171590 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.171681 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:55.172035 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.172552 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.174094 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.174481 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.174619 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.174672 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.174851 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.174924 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:34:55.175066 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.171180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40065 (local address 127.27.122.195:36117)
0928 16:34:55.171300 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:55.171316 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:55.171420 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:55.171724 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:55.171727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:55.171752 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:55.171893 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.171902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.172695 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.172700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.173953 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.173964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.174604 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.174608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.174726 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.174743 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 16:34:55.174833 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 16:34:55.174906 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":1}
I20260928 16:34:55.175071 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.171181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:43365 (local address 127.27.122.195:33303)
0928 16:34:55.171726 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:55.171738 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:55.171826 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:55.172052 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:55.172055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:55.172074 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:55.172214 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.172220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.173009 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.173013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.173875 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.173885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.174548 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.174551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.174658 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.174682 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:34:55.174780 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:34:55.174848 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":177,"thread_start_us":143,"threads_started":1}
I20260928 16:34:55.175834 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca3ed4f8a94ba090455eaae8eb32f2" candidate_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ef0a417149a4e8cb260476daacd441a" is_pre_election: true
I20260928 16:34:55.175836 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca3ed4f8a94ba090455eaae8eb32f2" candidate_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18222ddaedc74c41b6f47face5c2b261" is_pre_election: true
I20260928 16:34:55.176142 3544 raft_consensus.cc:2454] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e6c90c6b1fe4f209af81e4d18b9a38c in term 0.
I20260928 16:34:55.176167 3403 raft_consensus.cc:2454] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e6c90c6b1fe4f209af81e4d18b9a38c in term 0.
I20260928 16:34:55.176715 3615 leader_election.cc:302] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 18222ddaedc74c41b6f47face5c2b261, 4ef0a417149a4e8cb260476daacd441a, 5e6c90c6b1fe4f209af81e4d18b9a38c; no voters:
I20260928 16:34:55.177033 3767 raft_consensus.cc:2781] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:55.177132 3767 raft_consensus.cc:482] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:55.177197 3767 raft_consensus.cc:3037] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:55.178267 3767 raft_consensus.cc:504] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.178632 3767 leader_election.cc:288] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [CANDIDATE]: Term 1 election: Requested vote from peers 4ef0a417149a4e8cb260476daacd441a (127.27.122.194:43365), 18222ddaedc74c41b6f47face5c2b261 (127.27.122.193:40065)
I20260928 16:34:55.178997 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca3ed4f8a94ba090455eaae8eb32f2" candidate_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18222ddaedc74c41b6f47face5c2b261"
I20260928 16:34:55.179001 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ca3ed4f8a94ba090455eaae8eb32f2" candidate_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ef0a417149a4e8cb260476daacd441a"
I20260928 16:34:55.179157 3403 raft_consensus.cc:3037] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:55.179164 3544 raft_consensus.cc:3037] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:55.180224 3403 raft_consensus.cc:2454] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e6c90c6b1fe4f209af81e4d18b9a38c in term 1.
I20260928 16:34:55.180233 3544 raft_consensus.cc:2454] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e6c90c6b1fe4f209af81e4d18b9a38c in term 1.
I20260928 16:34:55.180574 3618 leader_election.cc:302] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18222ddaedc74c41b6f47face5c2b261, 5e6c90c6b1fe4f209af81e4d18b9a38c; no voters:
I20260928 16:34:55.180797 3767 raft_consensus.cc:2781] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:55.181084 3767 raft_consensus.cc:686] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 LEADER]: Becoming Leader. State: Replica: 5e6c90c6b1fe4f209af81e4d18b9a38c, State: Running, Role: LEADER
I20260928 16:34:55.181488 3767 consensus_queue.cc:237] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.184939 3243 catalog_manager.cc:5495] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195). New cstate: current_term: 1 leader_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } health_report { overall_health: HEALTHY } } }
W20260928 16:34:55.194023 3732 tablet.cc:2369] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:34:55.220234 3591 tablet.cc:2369] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:55.240859 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.237738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.27.122.195:41173)
0928 16:34:55.237870 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.237874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.237888 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:34:55.237965 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.237967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.238020 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.238120 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:55.238328 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.238830 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.239637 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.239852 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.239926 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.240149 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:55.240563 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:55.240650 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.240704 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.240753 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:34:55.246830 3403 raft_consensus.cc:1261] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Refusing update from remote peer 5e6c90c6b1fe4f209af81e4d18b9a38c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:55.246830 3544 raft_consensus.cc:1261] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Refusing update from remote peer 5e6c90c6b1fe4f209af81e4d18b9a38c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:55.247555 3767 consensus_queue.cc:1035] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [LEADER]: Connected to new peer: Peer: permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:55.247741 3769 consensus_queue.cc:1035] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:55.254532 3772 mvcc.cc:205] Tried to move back new op lower bound from 7334352057323319296 to 7334352057066541056. Current Snapshot: MvccSnapshot[applied={T|T < 7334352057323319296}]
I20260928 16:34:55.256843 3773 mvcc.cc:205] Tried to move back new op lower bound from 7334352057323319296 to 7334352057066541056. Current Snapshot: MvccSnapshot[applied={T|T < 7334352057323319296}]
I20260928 16:34:55.257103 3774 mvcc.cc:205] Tried to move back new op lower bound from 7334352057323319296 to 7334352057066541056. Current Snapshot: MvccSnapshot[applied={T|T < 7334352057323319296 or (T in {7334352057323319296})}]
I20260928 16:34:55.262013 3243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:55.262188 3243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:55.264694 3243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794: Sending DeleteTablet for 3 replicas of tablet 86ca3ed4f8a94ba090455eaae8eb32f2
I20260928 16:34:55.265421 3524 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ca3ed4f8a94ba090455eaae8eb32f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:55 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:40936
I20260928 16:34:55.265476 3383 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ca3ed4f8a94ba090455eaae8eb32f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:55 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:41496
I20260928 16:34:55.265558 3665 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ca3ed4f8a94ba090455eaae8eb32f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:55 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:47606
I20260928 16:34:55.266052 3781 tablet_replica.cc:330] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a: stopping tablet replica
I20260928 16:34:55.266376 3781 raft_consensus.cc:2229] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:55.266546 3783 tablet_replica.cc:330] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c: stopping tablet replica
I20260928 16:34:55.266829 3783 raft_consensus.cc:2229] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:55.266758 3243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:34:55.267161 3243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:55.267189 3783 raft_consensus.cc:2258] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:55.268047 3781 raft_consensus.cc:2258] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:55.269496 3783 ts_tablet_manager.cc:1905] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:55.270948 3781 ts_tablet_manager.cc:1905] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:55.272074 3524 tablet_service.cc:1467] Processing CreateTablet for tablet 509617b29f4d4146b385e7eb052e5480 (DEFAULT_TABLE table=test-table [id=8adaefd0d6f74a7f9600bb7fbff67421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:55.272181 3383 tablet_service.cc:1467] Processing CreateTablet for tablet 509617b29f4d4146b385e7eb052e5480 (DEFAULT_TABLE table=test-table [id=8adaefd0d6f74a7f9600bb7fbff67421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:55.272336 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 509617b29f4d4146b385e7eb052e5480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.272372 3665 tablet_service.cc:1467] Processing CreateTablet for tablet 509617b29f4d4146b385e7eb052e5480 (DEFAULT_TABLE table=test-table [id=8adaefd0d6f74a7f9600bb7fbff67421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:55.272454 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 509617b29f4d4146b385e7eb052e5480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.274921 3760 tablet_bootstrap.cc:492] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a: Bootstrap starting.
I20260928 16:34:55.274997 3759 tablet_bootstrap.cc:492] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261: Bootstrap starting.
I20260928 16:34:55.275213 3665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 509617b29f4d4146b385e7eb052e5480. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.275903 3760 tablet_bootstrap.cc:654] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.275911 3759 tablet_bootstrap.cc:654] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.277268 3781 ts_tablet_manager.cc:1918] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:55.277464 3781 log.cc:1196] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/86ca3ed4f8a94ba090455eaae8eb32f2
I20260928 16:34:55.277803 3781 ts_tablet_manager.cc:1939] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 4ef0a417149a4e8cb260476daacd441a: Deleting consensus metadata
I20260928 16:34:55.277872 3761 tablet_bootstrap.cc:492] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Bootstrap starting.
I20260928 16:34:55.278772 3761 tablet_bootstrap.cc:654] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.279116 3229 catalog_manager.cc:4841] TS 4ef0a417149a4e8cb260476daacd441a (127.27.122.194:43365): tablet 86ca3ed4f8a94ba090455eaae8eb32f2 (table test-table [id=5e220669c1f34a1ab54c41a16a0797b1]) successfully deleted
I20260928 16:34:55.279750 3759 tablet_bootstrap.cc:492] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261: No bootstrap required, opened a new log
I20260928 16:34:55.279834 3759 ts_tablet_manager.cc:1397] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 16:34:55.280262 3759 raft_consensus.cc:348] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.280367 3759 raft_consensus.cc:374] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.280397 3759 raft_consensus.cc:729] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18222ddaedc74c41b6f47face5c2b261, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.280516 3759 consensus_queue.cc:260] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.280758 3759 ts_tablet_manager.cc:1428] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:34:55.280822 3761 tablet_bootstrap.cc:492] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c: No bootstrap required, opened a new log
I20260928 16:34:55.280896 3761 ts_tablet_manager.cc:1397] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:34:55.281328 3761 raft_consensus.cc:348] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.281412 3761 raft_consensus.cc:374] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.281455 3761 raft_consensus.cc:729] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e6c90c6b1fe4f209af81e4d18b9a38c, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.281570 3761 consensus_queue.cc:260] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.281867 3761 ts_tablet_manager.cc:1428] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:55.282536 3782 tablet_replica.cc:330] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261: stopping tablet replica
I20260928 16:34:55.282892 3782 raft_consensus.cc:2229] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:55.283223 3782 raft_consensus.cc:2258] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:55.283687 3783 ts_tablet_manager.cc:1918] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:55.283785 3783 log.cc:1196] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/86ca3ed4f8a94ba090455eaae8eb32f2
I20260928 16:34:55.284128 3783 ts_tablet_manager.cc:1939] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Deleting consensus metadata
I20260928 16:34:55.285965 3230 catalog_manager.cc:4841] TS 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195:41173): tablet 86ca3ed4f8a94ba090455eaae8eb32f2 (table test-table [id=5e220669c1f34a1ab54c41a16a0797b1]) successfully deleted
I20260928 16:34:55.286725 3782 ts_tablet_manager.cc:1905] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:55.289587 3782 ts_tablet_manager.cc:1918] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:55.289705 3782 log.cc:1196] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/wal/wals/86ca3ed4f8a94ba090455eaae8eb32f2
I20260928 16:34:55.289714 3760 tablet_bootstrap.cc:492] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a: No bootstrap required, opened a new log
I20260928 16:34:55.289798 3760 ts_tablet_manager.cc:1397] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260928 16:34:55.290073 3782 ts_tablet_manager.cc:1939] T 86ca3ed4f8a94ba090455eaae8eb32f2 P 18222ddaedc74c41b6f47face5c2b261: Deleting consensus metadata
I20260928 16:34:55.290277 3760 raft_consensus.cc:348] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.290397 3760 raft_consensus.cc:374] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.290429 3760 raft_consensus.cc:729] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef0a417149a4e8cb260476daacd441a, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.290555 3760 consensus_queue.cc:260] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.290802 3760 ts_tablet_manager.cc:1428] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:55.291373 3232 catalog_manager.cc:4841] TS 18222ddaedc74c41b6f47face5c2b261 (127.27.122.193:40065): tablet 86ca3ed4f8a94ba090455eaae8eb32f2 (table test-table [id=5e220669c1f34a1ab54c41a16a0797b1]) successfully deleted
I20260928 16:34:55.294315 3765 raft_consensus.cc:482] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:55.294412 3765 raft_consensus.cc:504] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.295529 3765 leader_election.cc:288] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ef0a417149a4e8cb260476daacd441a (127.27.122.194:43365), 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195:41173)
I20260928 16:34:55.300010 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.295860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:35047 (local address 127.27.122.195:41173)
0928 16:34:55.295981 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.295986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.296400 (+ 414us) server_negotiation.cc:407] Connection header received
0928 16:34:55.296556 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.296559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.296599 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.296687 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:55.297022 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.297534 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.298691 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.299299 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.299373 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.299456 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.299811 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.299886 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 16:34:55.300578 3755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.295999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:46505 (local address 127.27.122.194:43365)
0928 16:34:55.296101 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.296105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.296210 (+ 105us) server_negotiation.cc:407] Connection header received
0928 16:34:55.296421 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.296425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.296475 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.296547 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:55.297012 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.297785 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.299457 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.300015 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.300156 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.300217 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.300405 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.300469 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 16:34:55.300627 3787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.295713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:43365 (local address 127.27.122.193:46505)
0928 16:34:55.296121 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:55.296136 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:55.296283 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:55.296663 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:55.296667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:55.296686 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:55.296858 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.296865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.297926 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.297929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.299323 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.299335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.300163 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.300167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.300269 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.300295 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:34:55.300377 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 16:34:55.300448 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":6,"spinlock_wait_cycles":15744,"thread_start_us":62,"threads_started":1}
I20260928 16:34:55.301095 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.295713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:41173 (local address 127.27.122.193:35047)
0928 16:34:55.296321 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:55.296336 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:55.296458 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:55.296698 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:55.296701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:55.296740 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:55.296884 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.296889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.297653 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.297656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.298510 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.298520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.299551 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.299554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.299673 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.299694 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:34:55.300333 (+ 639us) client_negotiation.cc:240] Negotiation successful
0928 16:34:55.300431 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":100,"spinlock_wait_cycles":768,"thread_start_us":70,"threads_started":1}
I20260928 16:34:55.301366 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "509617b29f4d4146b385e7eb052e5480" candidate_uuid: "18222ddaedc74c41b6f47face5c2b261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ef0a417149a4e8cb260476daacd441a" is_pre_election: true
I20260928 16:34:55.301502 3544 raft_consensus.cc:2454] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18222ddaedc74c41b6f47face5c2b261 in term 0.
I20260928 16:34:55.301702 3685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "509617b29f4d4146b385e7eb052e5480" candidate_uuid: "18222ddaedc74c41b6f47face5c2b261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" is_pre_election: true
I20260928 16:34:55.301927 3685 raft_consensus.cc:2454] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18222ddaedc74c41b6f47face5c2b261 in term 0.
I20260928 16:34:55.302390 3334 leader_election.cc:302] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18222ddaedc74c41b6f47face5c2b261, 5e6c90c6b1fe4f209af81e4d18b9a38c; no voters:
I20260928 16:34:55.302629 3765 raft_consensus.cc:2781] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:55.302690 3765 raft_consensus.cc:482] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:55.302729 3765 raft_consensus.cc:3037] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:55.303603 3765 raft_consensus.cc:504] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.303967 3765 leader_election.cc:288] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [CANDIDATE]: Term 1 election: Requested vote from peers 4ef0a417149a4e8cb260476daacd441a (127.27.122.194:43365), 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195:41173)
I20260928 16:34:55.304291 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "509617b29f4d4146b385e7eb052e5480" candidate_uuid: "18222ddaedc74c41b6f47face5c2b261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ef0a417149a4e8cb260476daacd441a"
I20260928 16:34:55.304324 3685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "509617b29f4d4146b385e7eb052e5480" candidate_uuid: "18222ddaedc74c41b6f47face5c2b261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c"
I20260928 16:34:55.304423 3685 raft_consensus.cc:3037] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:55.304423 3544 raft_consensus.cc:3037] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:55.305567 3544 raft_consensus.cc:2454] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18222ddaedc74c41b6f47face5c2b261 in term 1.
I20260928 16:34:55.305567 3685 raft_consensus.cc:2454] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18222ddaedc74c41b6f47face5c2b261 in term 1.
I20260928 16:34:55.305892 3334 leader_election.cc:302] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18222ddaedc74c41b6f47face5c2b261, 5e6c90c6b1fe4f209af81e4d18b9a38c; no voters:
I20260928 16:34:55.306087 3765 raft_consensus.cc:2781] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:55.306329 3765 raft_consensus.cc:686] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [term 1 LEADER]: Becoming Leader. State: Replica: 18222ddaedc74c41b6f47face5c2b261, State: Running, Role: LEADER
I20260928 16:34:55.306569 3765 consensus_queue.cc:237] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.309175 3243 catalog_manager.cc:5495] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18222ddaedc74c41b6f47face5c2b261 (127.27.122.193). New cstate: current_term: 1 leader_uuid: "18222ddaedc74c41b6f47face5c2b261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } health_report { overall_health: UNKNOWN } } }
Sep 28 16:34:55 dist-test-slave-g9sh krb5kdc[3200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613294, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.254@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
I20260928 16:34:55.348141 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.338700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:45455 (local address 127.0.0.1:59264)
0928 16:34:55.339037 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:55.339053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:55.339189 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:55.339442 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:55.339445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:55.339977 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:55.340125 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.340133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.340870 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.340874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.341801 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.341812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.342241 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.342245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.342365 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.342945 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:55.342961 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:55.344545 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:55.346510 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:55.346527 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:55.346532 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:55.346844 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:55.347138 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:55.347140 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:55.347141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:55.347223 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:55.347642 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:55.347648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:55.347785 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 16:34:55.347890 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 16:34:55.348002 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":130,"threads_started":1}
I20260928 16:34:55.348276 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.338772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.27.122.254:45455)
0928 16:34:55.338921 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.338926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.339151 (+ 225us) server_negotiation.cc:407] Connection header received
0928 16:34:55.339282 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.339284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.339324 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.339391 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:55.340245 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.340759 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.341915 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.342133 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.342211 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.344689 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:55.344693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:55.344696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:55.344749 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:55.346361 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:55.346939 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:55.346943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:55.346945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:55.347034 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:55.347309 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:55.347312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:55.347314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:55.347519 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:55.347617 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.347968 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.348092 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:34:55.352289 3243 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:59264:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 16:34:55.353122 3243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:55.357950 3383 tablet_service.cc:1467] Processing CreateTablet for tablet 6309347cd5f740d1a399c6f8ea9a4d14 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c7677c8627e4190aa9db5b594b9ffe4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:34:55.358067 3665 tablet_service.cc:1467] Processing CreateTablet for tablet 6309347cd5f740d1a399c6f8ea9a4d14 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c7677c8627e4190aa9db5b594b9ffe4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:34:55.358037 3524 tablet_service.cc:1467] Processing CreateTablet for tablet 6309347cd5f740d1a399c6f8ea9a4d14 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c7677c8627e4190aa9db5b594b9ffe4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 16:34:55.358402 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6309347cd5f740d1a399c6f8ea9a4d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.358449 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6309347cd5f740d1a399c6f8ea9a4d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.358449 3665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6309347cd5f740d1a399c6f8ea9a4d14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.361187 3761 tablet_bootstrap.cc:492] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Bootstrap starting.
I20260928 16:34:55.361267 3759 tablet_bootstrap.cc:492] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261: Bootstrap starting.
I20260928 16:34:55.361466 3760 tablet_bootstrap.cc:492] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a: Bootstrap starting.
I20260928 16:34:55.362114 3761 tablet_bootstrap.cc:654] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.362178 3759 tablet_bootstrap.cc:654] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.362464 3760 tablet_bootstrap.cc:654] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.363919 3759 tablet_bootstrap.cc:492] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261: No bootstrap required, opened a new log
I20260928 16:34:55.363915 3760 tablet_bootstrap.cc:492] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a: No bootstrap required, opened a new log
I20260928 16:34:55.364006 3760 ts_tablet_manager.cc:1397] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:34:55.364007 3759 ts_tablet_manager.cc:1397] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:34:55.364445 3760 raft_consensus.cc:348] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.364445 3759 raft_consensus.cc:348] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.364566 3760 raft_consensus.cc:374] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.364566 3759 raft_consensus.cc:374] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.364602 3760 raft_consensus.cc:729] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef0a417149a4e8cb260476daacd441a, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.364624 3759 raft_consensus.cc:729] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18222ddaedc74c41b6f47face5c2b261, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.364735 3760 consensus_queue.cc:260] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.364751 3759 consensus_queue.cc:260] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.365036 3759 ts_tablet_manager.cc:1428] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:55.365046 3766 tablet_replica.cc:439] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } } }
I20260928 16:34:55.365192 3766 tablet_replica.cc:442] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:55.365223 3765 tablet_replica.cc:439] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } } }
I20260928 16:34:55.365330 3765 tablet_replica.cc:442] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:55.365427 3760 ts_tablet_manager.cc:1428] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:55.366230 3761 tablet_bootstrap.cc:492] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c: No bootstrap required, opened a new log
I20260928 16:34:55.366321 3761 ts_tablet_manager.cc:1397] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 16:34:55.367126 3761 raft_consensus.cc:348] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.367246 3761 raft_consensus.cc:374] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.367292 3761 raft_consensus.cc:729] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e6c90c6b1fe4f209af81e4d18b9a38c, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.367429 3761 consensus_queue.cc:260] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.367731 3761 ts_tablet_manager.cc:1428] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 16:34:55.367674 3767 tablet_replica.cc:439] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } } }
I20260928 16:34:55.368999 3767 tablet_replica.cc:442] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 16:34:55.508275 3450 tablet.cc:2369] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:34:55.585914 3766 raft_consensus.cc:482] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:55.586068 3766 raft_consensus.cc:504] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.587273 3766 leader_election.cc:288] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18222ddaedc74c41b6f47face5c2b261 (127.27.122.193:40065), 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195:41173)
I20260928 16:34:55.591630 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.587749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:42005 (local address 127.27.122.193:40065)
0928 16:34:55.587879 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.587886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.588216 (+ 330us) server_negotiation.cc:407] Connection header received
0928 16:34:55.588363 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.588366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.588408 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.588506 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:55.588850 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.589370 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.590505 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.591033 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.591133 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.591216 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.591402 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.591485 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:34:55.591666 3757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.587684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:55463 (local address 127.27.122.195:41173)
0928 16:34:55.587835 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.587839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.588092 (+ 253us) server_negotiation.cc:407] Connection header received
0928 16:34:55.588304 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.588309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.588367 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.588435 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:34:55.588803 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.589387 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.590532 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.591098 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.591217 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.591269 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.591475 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.591557 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 16:34:55.591648 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.587549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40065 (local address 127.27.122.194:42005)
0928 16:34:55.588113 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:55.588125 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:55.588256 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:55.588543 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:55.588547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:55.588570 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:55.588707 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.588712 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.589482 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.589486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.590375 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.590384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.591151 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.591158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.591262 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.591292 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:34:55.591360 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 16:34:55.591435 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":103,"thread_start_us":108,"threads_started":1}
I20260928 16:34:55.591655 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.587549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:41173 (local address 127.27.122.194:55463)
0928 16:34:55.587989 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:55.588001 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:55.588140 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:55.588479 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:55.588483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:55.588504 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:34:55.588683 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.588690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.589534 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.589536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.590384 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.590393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.591223 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.591229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.591323 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.591362 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 16:34:55.591443 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 16:34:55.591505 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":67,"thread_start_us":136,"threads_started":1}
I20260928 16:34:55.592109 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6309347cd5f740d1a399c6f8ea9a4d14" candidate_uuid: "4ef0a417149a4e8cb260476daacd441a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18222ddaedc74c41b6f47face5c2b261" is_pre_election: true
I20260928 16:34:55.592296 3403 raft_consensus.cc:2454] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ef0a417149a4e8cb260476daacd441a in term 0.
I20260928 16:34:55.592329 3685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6309347cd5f740d1a399c6f8ea9a4d14" candidate_uuid: "4ef0a417149a4e8cb260476daacd441a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" is_pre_election: true
I20260928 16:34:55.592463 3685 raft_consensus.cc:2454] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ef0a417149a4e8cb260476daacd441a in term 0.
I20260928 16:34:55.592727 3477 leader_election.cc:302] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18222ddaedc74c41b6f47face5c2b261, 4ef0a417149a4e8cb260476daacd441a; no voters:
I20260928 16:34:55.593039 3766 raft_consensus.cc:2781] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:55.593110 3766 raft_consensus.cc:482] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:55.593149 3766 raft_consensus.cc:3037] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:55.594326 3766 raft_consensus.cc:504] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.594723 3766 leader_election.cc:288] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [CANDIDATE]: Term 1 election: Requested vote from peers 18222ddaedc74c41b6f47face5c2b261 (127.27.122.193:40065), 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195:41173)
I20260928 16:34:55.595074 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6309347cd5f740d1a399c6f8ea9a4d14" candidate_uuid: "4ef0a417149a4e8cb260476daacd441a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18222ddaedc74c41b6f47face5c2b261"
I20260928 16:34:55.595213 3403 raft_consensus.cc:3037] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:55.595167 3685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6309347cd5f740d1a399c6f8ea9a4d14" candidate_uuid: "4ef0a417149a4e8cb260476daacd441a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c"
I20260928 16:34:55.595323 3685 raft_consensus.cc:3037] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:55.596350 3403 raft_consensus.cc:2454] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ef0a417149a4e8cb260476daacd441a in term 1.
I20260928 16:34:55.596598 3685 raft_consensus.cc:2454] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ef0a417149a4e8cb260476daacd441a in term 1.
I20260928 16:34:55.596863 3477 leader_election.cc:302] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18222ddaedc74c41b6f47face5c2b261, 4ef0a417149a4e8cb260476daacd441a; no voters:
I20260928 16:34:55.597093 3766 raft_consensus.cc:2781] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:55.597324 3766 raft_consensus.cc:686] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [term 1 LEADER]: Becoming Leader. State: Replica: 4ef0a417149a4e8cb260476daacd441a, State: Running, Role: LEADER
I20260928 16:34:55.597589 3766 consensus_queue.cc:237] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.598742 3802 tablet_replica.cc:439] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a: TxnStatusTablet state changed. Reason: New leader 4ef0a417149a4e8cb260476daacd441a. Latest consensus state: current_term: 1 leader_uuid: "4ef0a417149a4e8cb260476daacd441a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } } }
I20260928 16:34:55.598855 3802 tablet_replica.cc:442] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:34:55.599155 3804 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 16:34:55.599326 3804 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 16:34:55.600087 3245 catalog_manager.cc:5495] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ef0a417149a4e8cb260476daacd441a (127.27.122.194). New cstate: current_term: 1 leader_uuid: "4ef0a417149a4e8cb260476daacd441a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } health_report { overall_health: UNKNOWN } } }
I20260928 16:34:55.662930 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.659140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:43365 (local address 127.0.0.1:40946)
0928 16:34:55.659281 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:55.659298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:55.659411 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:55.659781 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:55.659784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:55.659792 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:34:55.659909 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.659915 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.660646 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.660650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.661331 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.661340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.661799 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.661803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.661897 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.661939 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:34:55.662635 (+ 696us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:34:55.662652 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 16:34:55.662722 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 16:34:55.662805 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 16:34:55.662930 3755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.659266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40946 (local address 127.27.122.194:43365)
0928 16:34:55.659388 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.659393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.659407 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:34:55.659587 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.659589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.659660 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.659765 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:55.660020 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.660528 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.661453 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.661685 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.661801 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.662105 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:55.662527 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:55.662663 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.662731 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.662786 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:34:55.665985 3403 raft_consensus.cc:1261] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Refusing update from remote peer 4ef0a417149a4e8cb260476daacd441a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:55.666177 3685 raft_consensus.cc:1261] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Refusing update from remote peer 4ef0a417149a4e8cb260476daacd441a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:55.666592 3802 consensus_queue.cc:1035] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:55.666755 3766 consensus_queue.cc:1035] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:55.668498 3767 tablet_replica.cc:439] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c: TxnStatusTablet state changed. Reason: New leader 4ef0a417149a4e8cb260476daacd441a. Latest consensus state: current_term: 1 leader_uuid: "4ef0a417149a4e8cb260476daacd441a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } } }
I20260928 16:34:55.668632 3767 tablet_replica.cc:442] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:55.668556 3765 tablet_replica.cc:439] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261: TxnStatusTablet state changed. Reason: New leader 4ef0a417149a4e8cb260476daacd441a. Latest consensus state: current_term: 1 leader_uuid: "4ef0a417149a4e8cb260476daacd441a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } } }
I20260928 16:34:55.668681 3765 tablet_replica.cc:442] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:55.671291 3809 tablet_replica.cc:439] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ef0a417149a4e8cb260476daacd441a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } } }
I20260928 16:34:55.671432 3809 tablet_replica.cc:442] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:34:55.671569 3767 tablet_replica.cc:439] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ef0a417149a4e8cb260476daacd441a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } } }
I20260928 16:34:55.671692 3767 tablet_replica.cc:442] T 6309347cd5f740d1a399c6f8ea9a4d14 P 5e6c90c6b1fe4f209af81e4d18b9a38c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:55.671922 3766 tablet_replica.cc:439] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ef0a417149a4e8cb260476daacd441a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } } }
I20260928 16:34:55.672021 3766 tablet_replica.cc:442] T 6309347cd5f740d1a399c6f8ea9a4d14 P 4ef0a417149a4e8cb260476daacd441a: This TxnStatusTablet replica's current role is: LEADER
I20260928 16:34:55.672415 3765 tablet_replica.cc:439] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ef0a417149a4e8cb260476daacd441a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } } }
I20260928 16:34:55.672530 3765 tablet_replica.cc:442] T 6309347cd5f740d1a399c6f8ea9a4d14 P 18222ddaedc74c41b6f47face5c2b261: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 16:34:55.680536 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.676276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41506 (local address 127.27.122.193:40065)
0928 16:34:55.676382 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.676386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.676511 (+ 125us) server_negotiation.cc:407] Connection header received
0928 16:34:55.676713 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.676716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.676779 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.676883 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:55.677187 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.678036 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.678834 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.679232 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.679361 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.679620 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:55.680157 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:55.680267 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.680326 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.680386 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 16:34:55.691462 3755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.687713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:50355 (local address 127.27.122.194:43365)
0928 16:34:55.687871 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.687876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.688082 (+ 206us) server_negotiation.cc:407] Connection header received
0928 16:34:55.688191 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.688193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.688237 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.688299 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:55.688544 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.689097 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.690344 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.690557 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.690639 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.691010 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:55.691089 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:55.691172 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.691290 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.691338 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:34:55.691545 3814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.687608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:43365 (local address 127.27.122.193:50355)
0928 16:34:55.687996 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:55.688012 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:55.688113 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:55.688333 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:55.688336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:55.688343 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:34:55.688460 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.688465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.689192 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.689195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.690222 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.690232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.690675 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.690679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.690809 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.690862 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:34:55.691197 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:34:55.691204 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 16:34:55.691281 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:34:55.691360 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":164,"threads_started":1}
I20260928 16:34:55.698256 3685 raft_consensus.cc:1261] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Refusing update from remote peer 18222ddaedc74c41b6f47face5c2b261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:55.698375 3544 raft_consensus.cc:1261] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Refusing update from remote peer 18222ddaedc74c41b6f47face5c2b261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:55.698935 3789 consensus_queue.cc:1035] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:55.699365 3765 consensus_queue.cc:1035] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:55.735766 3818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.731783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40065 (local address 127.27.122.194:59883)
0928 16:34:55.732108 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:55.732124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:55.732289 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:55.732717 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:55.732722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:55.732734 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 16:34:55.732880 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.732887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.733868 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.733870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.734505 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:55.734512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.735001 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.735005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:55.735114 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.735162 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 16:34:55.735475 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 16:34:55.735484 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:34:55.735549 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 16:34:55.735640 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":116,"threads_started":1}
I20260928 16:34:55.735757 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.731860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:59883 (local address 127.27.122.193:40065)
0928 16:34:55.732001 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.732006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.732195 (+ 189us) server_negotiation.cc:407] Connection header received
0928 16:34:55.732425 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.732430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.732489 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.732543 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:55.733033 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.733754 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.734645 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.734886 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.734998 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.735304 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:55.735388 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:55.735469 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.735560 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.735610 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:34:55.773749 3243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:55.773931 3243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:55.776178 3243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794: Sending DeleteTablet for 3 replicas of tablet 509617b29f4d4146b385e7eb052e5480
I20260928 16:34:55.776814 3524 tablet_service.cc:1514] Processing DeleteTablet for tablet 509617b29f4d4146b385e7eb052e5480 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:55 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:40936
I20260928 16:34:55.777021 3665 tablet_service.cc:1514] Processing DeleteTablet for tablet 509617b29f4d4146b385e7eb052e5480 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:55 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:47606
I20260928 16:34:55.777168 3781 tablet_replica.cc:330] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a: stopping tablet replica
I20260928 16:34:55.777266 3783 tablet_replica.cc:330] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c: stopping tablet replica
I20260928 16:34:55.777300 3781 raft_consensus.cc:2229] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:55.777412 3783 raft_consensus.cc:2229] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:55.777431 3781 raft_consensus.cc:2258] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:55.777469 3383 tablet_service.cc:1514] Processing DeleteTablet for tablet 509617b29f4d4146b385e7eb052e5480 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:55 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:41496
I20260928 16:34:55.777544 3783 raft_consensus.cc:2258] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:55.777660 3782 tablet_replica.cc:330] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261: stopping tablet replica
I20260928 16:34:55.777789 3782 raft_consensus.cc:2229] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:55.778072 3782 raft_consensus.cc:2258] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:55.778226 3781 ts_tablet_manager.cc:1905] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:55.778412 3783 ts_tablet_manager.cc:1905] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:55.778980 3782 ts_tablet_manager.cc:1905] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:55.781299 3781 ts_tablet_manager.cc:1918] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:55.781369 3781 log.cc:1196] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/509617b29f4d4146b385e7eb052e5480
I20260928 16:34:55.781653 3781 ts_tablet_manager.cc:1939] T 509617b29f4d4146b385e7eb052e5480 P 4ef0a417149a4e8cb260476daacd441a: Deleting consensus metadata
I20260928 16:34:55.781597 3783 ts_tablet_manager.cc:1918] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:55.781713 3783 log.cc:1196] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/509617b29f4d4146b385e7eb052e5480
I20260928 16:34:55.782202 3783 ts_tablet_manager.cc:1939] T 509617b29f4d4146b385e7eb052e5480 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Deleting consensus metadata
I20260928 16:34:55.782565 3229 catalog_manager.cc:4841] TS 4ef0a417149a4e8cb260476daacd441a (127.27.122.194:43365): tablet 509617b29f4d4146b385e7eb052e5480 (table test-table [id=8adaefd0d6f74a7f9600bb7fbff67421]) successfully deleted
I20260928 16:34:55.783034 3230 catalog_manager.cc:4841] TS 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195:41173): tablet 509617b29f4d4146b385e7eb052e5480 (table test-table [id=8adaefd0d6f74a7f9600bb7fbff67421]) successfully deleted
I20260928 16:34:55.786800 3782 ts_tablet_manager.cc:1918] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:55.786899 3782 log.cc:1196] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/wal/wals/509617b29f4d4146b385e7eb052e5480
I20260928 16:34:55.787149 3782 ts_tablet_manager.cc:1939] T 509617b29f4d4146b385e7eb052e5480 P 18222ddaedc74c41b6f47face5c2b261: Deleting consensus metadata
I20260928 16:34:55.787814 3232 catalog_manager.cc:4841] TS 18222ddaedc74c41b6f47face5c2b261 (127.27.122.193:40065): tablet 509617b29f4d4146b385e7eb052e5480 (table test-table [id=8adaefd0d6f74a7f9600bb7fbff67421]) successfully deleted
Sep 28 16:34:55 dist-test-slave-g9sh krb5kdc[3200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 16:34:55 dist-test-slave-g9sh krb5kdc[3200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:34:55 dist-test-slave-g9sh krb5kdc[3200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:34:55.810367 3294 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:34:55.803564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.27.122.254:45455)
0928 16:34:55.803715 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.803720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.803813 (+ 93us) server_negotiation.cc:407] Connection header received
0928 16:34:55.803933 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.803937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.803992 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.804090 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:55.804762 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.805639 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.806502 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.806881 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.806994 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.810234 (+ 3240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59272: BlockingRecv error: recv got EOF from 127.0.0.1:59272 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 16:34:55 dist-test-slave-g9sh krb5kdc[3200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM
I20260928 16:34:55.822384 3294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.811925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.27.122.254:45455)
0928 16:34:55.812056 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.812060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.812434 (+ 374us) server_negotiation.cc:407] Connection header received
0928 16:34:55.812510 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.812513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.812559 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.812643 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:55.813271 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.814082 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.814825 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.815163 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.815266 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.817824 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:55.817828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:55.817831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:55.817862 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:55.819428 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:55.819821 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:55.819824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:55.819826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:55.819917 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:55.820176 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:55.820179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:55.820181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:55.820389 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:55.820611 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.822090 (+ 1479us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.822212 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:34:55.825220 3243 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:34:55.825726 3243 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:55.830878 3383 tablet_service.cc:1467] Processing CreateTablet for tablet cf3135eb8163429496e27824452d82e9 (DEFAULT_TABLE table=test-table [id=5c3f1e49523645fba58b9a1e82cfe249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:55.830986 3665 tablet_service.cc:1467] Processing CreateTablet for tablet cf3135eb8163429496e27824452d82e9 (DEFAULT_TABLE table=test-table [id=5c3f1e49523645fba58b9a1e82cfe249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:55.831208 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3135eb8163429496e27824452d82e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.831261 3665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3135eb8163429496e27824452d82e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.833231 3524 tablet_service.cc:1467] Processing CreateTablet for tablet cf3135eb8163429496e27824452d82e9 (DEFAULT_TABLE table=test-table [id=5c3f1e49523645fba58b9a1e82cfe249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:55.833524 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3135eb8163429496e27824452d82e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.833731 3759 tablet_bootstrap.cc:492] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261: Bootstrap starting.
I20260928 16:34:55.833806 3761 tablet_bootstrap.cc:492] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Bootstrap starting.
I20260928 16:34:55.834705 3761 tablet_bootstrap.cc:654] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.834705 3759 tablet_bootstrap.cc:654] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.835832 3760 tablet_bootstrap.cc:492] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a: Bootstrap starting.
I20260928 16:34:55.836143 3761 tablet_bootstrap.cc:492] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c: No bootstrap required, opened a new log
I20260928 16:34:55.836234 3761 ts_tablet_manager.cc:1397] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 16:34:55.836735 3760 tablet_bootstrap.cc:654] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.836688 3761 raft_consensus.cc:348] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.836804 3761 raft_consensus.cc:374] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.836845 3761 raft_consensus.cc:729] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e6c90c6b1fe4f209af81e4d18b9a38c, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.836997 3761 consensus_queue.cc:260] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.837318 3761 ts_tablet_manager.cc:1428] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:55.837924 3760 tablet_bootstrap.cc:492] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a: No bootstrap required, opened a new log
I20260928 16:34:55.838013 3760 ts_tablet_manager.cc:1397] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 16:34:55.838297 3759 tablet_bootstrap.cc:492] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261: No bootstrap required, opened a new log
I20260928 16:34:55.838390 3759 ts_tablet_manager.cc:1397] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 16:34:55.838425 3760 raft_consensus.cc:348] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.838551 3760 raft_consensus.cc:374] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.838589 3760 raft_consensus.cc:729] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef0a417149a4e8cb260476daacd441a, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.838713 3760 consensus_queue.cc:260] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.838876 3759 raft_consensus.cc:348] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.838967 3760 ts_tablet_manager.cc:1428] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:34:55.838984 3759 raft_consensus.cc:374] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.839016 3759 raft_consensus.cc:729] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18222ddaedc74c41b6f47face5c2b261, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.839154 3759 consensus_queue.cc:260] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.839411 3759 ts_tablet_manager.cc:1428] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:55.926954 3810 raft_consensus.cc:482] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:55.927093 3810 raft_consensus.cc:504] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.927641 3810 leader_election.cc:288] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18222ddaedc74c41b6f47face5c2b261 (127.27.122.193:40065), 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195:41173)
I20260928 16:34:55.928114 3684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3135eb8163429496e27824452d82e9" candidate_uuid: "4ef0a417149a4e8cb260476daacd441a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" is_pre_election: true
I20260928 16:34:55.928123 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3135eb8163429496e27824452d82e9" candidate_uuid: "4ef0a417149a4e8cb260476daacd441a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18222ddaedc74c41b6f47face5c2b261" is_pre_election: true
I20260928 16:34:55.928288 3684 raft_consensus.cc:2454] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ef0a417149a4e8cb260476daacd441a in term 0.
I20260928 16:34:55.928293 3403 raft_consensus.cc:2454] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ef0a417149a4e8cb260476daacd441a in term 0.
I20260928 16:34:55.928606 3477 leader_election.cc:302] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18222ddaedc74c41b6f47face5c2b261, 4ef0a417149a4e8cb260476daacd441a; no voters:
I20260928 16:34:55.928798 3810 raft_consensus.cc:2781] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:55.928879 3810 raft_consensus.cc:482] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:55.928956 3810 raft_consensus.cc:3037] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:55.930004 3810 raft_consensus.cc:504] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.930393 3810 leader_election.cc:288] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [CANDIDATE]: Term 1 election: Requested vote from peers 18222ddaedc74c41b6f47face5c2b261 (127.27.122.193:40065), 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195:41173)
I20260928 16:34:55.930748 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3135eb8163429496e27824452d82e9" candidate_uuid: "4ef0a417149a4e8cb260476daacd441a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18222ddaedc74c41b6f47face5c2b261"
I20260928 16:34:55.930780 3684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3135eb8163429496e27824452d82e9" candidate_uuid: "4ef0a417149a4e8cb260476daacd441a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c"
I20260928 16:34:55.930876 3403 raft_consensus.cc:3037] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:55.930920 3684 raft_consensus.cc:3037] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:55.931910 3684 raft_consensus.cc:2454] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ef0a417149a4e8cb260476daacd441a in term 1.
I20260928 16:34:55.931998 3403 raft_consensus.cc:2454] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ef0a417149a4e8cb260476daacd441a in term 1.
I20260928 16:34:55.932276 3475 leader_election.cc:302] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ef0a417149a4e8cb260476daacd441a, 5e6c90c6b1fe4f209af81e4d18b9a38c; no voters:
I20260928 16:34:55.932499 3810 raft_consensus.cc:2781] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:55.932626 3810 raft_consensus.cc:686] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [term 1 LEADER]: Becoming Leader. State: Replica: 4ef0a417149a4e8cb260476daacd441a, State: Running, Role: LEADER
I20260928 16:34:55.932780 3810 consensus_queue.cc:237] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.934628 3244 catalog_manager.cc:5495] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ef0a417149a4e8cb260476daacd441a (127.27.122.194). New cstate: current_term: 1 leader_uuid: "4ef0a417149a4e8cb260476daacd441a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } health_report { overall_health: UNKNOWN } } }
I20260928 16:34:55.964032 3755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:55.960728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.27.122.194:43365)
0928 16:34:55.960865 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:55.960868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:55.960882 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:34:55.960974 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:55.960976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:55.961020 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:55.961085 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:55.961342 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.961843 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.963035 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:55.963240 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:55.963314 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:55.963586 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:55.963651 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:55.963703 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:55.963826 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 16:34:55.963873 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:34:55.966517 3684 raft_consensus.cc:1261] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Refusing update from remote peer 4ef0a417149a4e8cb260476daacd441a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:55.966548 3403 raft_consensus.cc:1261] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Refusing update from remote peer 4ef0a417149a4e8cb260476daacd441a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:55.967074 3810 consensus_queue.cc:1035] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:55.967262 3766 consensus_queue.cc:1035] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [LEADER]: Connected to new peer: Peer: permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:55.977090 3244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:55.977237 3244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:55.979141 3244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794: Sending DeleteTablet for 3 replicas of tablet cf3135eb8163429496e27824452d82e9
I20260928 16:34:55.979655 3383 tablet_service.cc:1514] Processing DeleteTablet for tablet cf3135eb8163429496e27824452d82e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:55 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:41496
I20260928 16:34:55.979759 3665 tablet_service.cc:1514] Processing DeleteTablet for tablet cf3135eb8163429496e27824452d82e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:55 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:47606
I20260928 16:34:55.979847 3524 tablet_service.cc:1514] Processing DeleteTablet for tablet cf3135eb8163429496e27824452d82e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:55 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:40936
I20260928 16:34:55.979933 3783 tablet_replica.cc:330] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c: stopping tablet replica
I20260928 16:34:55.980010 3782 tablet_replica.cc:330] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261: stopping tablet replica
I20260928 16:34:55.980065 3783 raft_consensus.cc:2229] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:55.980137 3782 raft_consensus.cc:2229] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:55.980194 3783 raft_consensus.cc:2258] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:55.980257 3782 raft_consensus.cc:2258] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:55.980415 3781 tablet_replica.cc:330] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a: stopping tablet replica
I20260928 16:34:55.980530 3781 raft_consensus.cc:2229] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:55.980798 3781 raft_consensus.cc:2258] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:55.981045 3783 ts_tablet_manager.cc:1905] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:55.981527 3781 ts_tablet_manager.cc:1905] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:55.983650 3783 ts_tablet_manager.cc:1918] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:55.983718 3783 log.cc:1196] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/cf3135eb8163429496e27824452d82e9
I20260928 16:34:55.983783 3781 ts_tablet_manager.cc:1918] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:55.983839 3781 log.cc:1196] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/cf3135eb8163429496e27824452d82e9
I20260928 16:34:55.983969 3783 ts_tablet_manager.cc:1939] T cf3135eb8163429496e27824452d82e9 P 5e6c90c6b1fe4f209af81e4d18b9a38c: Deleting consensus metadata
I20260928 16:34:55.984017 3781 ts_tablet_manager.cc:1939] T cf3135eb8163429496e27824452d82e9 P 4ef0a417149a4e8cb260476daacd441a: Deleting consensus metadata
I20260928 16:34:55.984591 3230 catalog_manager.cc:4841] TS 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195:41173): tablet cf3135eb8163429496e27824452d82e9 (table test-table [id=5c3f1e49523645fba58b9a1e82cfe249]) successfully deleted
I20260928 16:34:55.984707 3782 ts_tablet_manager.cc:1905] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:55.985004 3229 catalog_manager.cc:4841] TS 4ef0a417149a4e8cb260476daacd441a (127.27.122.194:43365): tablet cf3135eb8163429496e27824452d82e9 (table test-table [id=5c3f1e49523645fba58b9a1e82cfe249]) successfully deleted
I20260928 16:34:55.985044 3245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:34:55.985447 3245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:34:55.986933 3782 ts_tablet_manager.cc:1918] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:34:55.987020 3782 log.cc:1196] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-0/wal/wals/cf3135eb8163429496e27824452d82e9
I20260928 16:34:55.987283 3782 ts_tablet_manager.cc:1939] T cf3135eb8163429496e27824452d82e9 P 18222ddaedc74c41b6f47face5c2b261: Deleting consensus metadata
I20260928 16:34:55.988061 3232 catalog_manager.cc:4841] TS 18222ddaedc74c41b6f47face5c2b261 (127.27.122.193:40065): tablet cf3135eb8163429496e27824452d82e9 (table test-table [id=5c3f1e49523645fba58b9a1e82cfe249]) successfully deleted
I20260928 16:34:55.990298 3383 tablet_service.cc:1467] Processing CreateTablet for tablet 59eb8ca24376470ba80cfef02b53faca (DEFAULT_TABLE table=test-table [id=e0904b5601be4a4fafcbaf0cf247d5f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:55.990327 3665 tablet_service.cc:1467] Processing CreateTablet for tablet 59eb8ca24376470ba80cfef02b53faca (DEFAULT_TABLE table=test-table [id=e0904b5601be4a4fafcbaf0cf247d5f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:55.990556 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59eb8ca24376470ba80cfef02b53faca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.990556 3665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59eb8ca24376470ba80cfef02b53faca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.992754 3759 tablet_bootstrap.cc:492] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261: Bootstrap starting.
I20260928 16:34:55.992785 3761 tablet_bootstrap.cc:492] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c: Bootstrap starting.
I20260928 16:34:55.992944 3524 tablet_service.cc:1467] Processing CreateTablet for tablet 59eb8ca24376470ba80cfef02b53faca (DEFAULT_TABLE table=test-table [id=e0904b5601be4a4fafcbaf0cf247d5f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:34:55.993198 3524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59eb8ca24376470ba80cfef02b53faca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:55.993595 3759 tablet_bootstrap.cc:654] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.993655 3761 tablet_bootstrap.cc:654] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.994876 3761 tablet_bootstrap.cc:492] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c: No bootstrap required, opened a new log
I20260928 16:34:55.994976 3761 ts_tablet_manager.cc:1397] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:55.995385 3761 raft_consensus.cc:348] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.995496 3761 raft_consensus.cc:374] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.995533 3761 raft_consensus.cc:729] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e6c90c6b1fe4f209af81e4d18b9a38c, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.995576 3760 tablet_bootstrap.cc:492] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a: Bootstrap starting.
I20260928 16:34:55.995667 3761 consensus_queue.cc:260] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.995957 3761 ts_tablet_manager.cc:1428] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:55.996376 3760 tablet_bootstrap.cc:654] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:55.997304 3759 tablet_bootstrap.cc:492] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261: No bootstrap required, opened a new log
I20260928 16:34:55.997378 3759 ts_tablet_manager.cc:1397] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 16:34:55.997748 3759 raft_consensus.cc:348] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.997848 3759 raft_consensus.cc:374] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:55.997879 3759 raft_consensus.cc:729] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18222ddaedc74c41b6f47face5c2b261, State: Initialized, Role: FOLLOWER
I20260928 16:34:55.997995 3759 consensus_queue.cc:260] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:55.998253 3759 ts_tablet_manager.cc:1428] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:34:55.999738 3760 tablet_bootstrap.cc:492] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a: No bootstrap required, opened a new log
I20260928 16:34:55.999817 3760 ts_tablet_manager.cc:1397] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 16:34:56.000187 3760 raft_consensus.cc:348] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:56.000284 3760 raft_consensus.cc:374] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:56.000314 3760 raft_consensus.cc:729] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef0a417149a4e8cb260476daacd441a, State: Initialized, Role: FOLLOWER
I20260928 16:34:56.000444 3760 consensus_queue.cc:260] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:56.000771 3760 ts_tablet_manager.cc:1428] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 16:34:56.198863 3819 raft_consensus.cc:482] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:34:56.199014 3819 raft_consensus.cc:504] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:56.199414 3819 leader_election.cc:288] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ef0a417149a4e8cb260476daacd441a (127.27.122.194:43365), 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195:41173)
I20260928 16:34:56.199918 3684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59eb8ca24376470ba80cfef02b53faca" candidate_uuid: "18222ddaedc74c41b6f47face5c2b261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" is_pre_election: true
I20260928 16:34:56.199941 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59eb8ca24376470ba80cfef02b53faca" candidate_uuid: "18222ddaedc74c41b6f47face5c2b261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ef0a417149a4e8cb260476daacd441a" is_pre_election: true
I20260928 16:34:56.200057 3684 raft_consensus.cc:2454] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18222ddaedc74c41b6f47face5c2b261 in term 0.
I20260928 16:34:56.200162 3544 raft_consensus.cc:2454] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18222ddaedc74c41b6f47face5c2b261 in term 0.
I20260928 16:34:56.200419 3334 leader_election.cc:302] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18222ddaedc74c41b6f47face5c2b261, 5e6c90c6b1fe4f209af81e4d18b9a38c; no voters:
I20260928 16:34:56.200628 3819 raft_consensus.cc:2781] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:34:56.200701 3819 raft_consensus.cc:482] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:34:56.200733 3819 raft_consensus.cc:3037] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:56.201831 3819 raft_consensus.cc:504] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:56.202162 3819 leader_election.cc:288] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [CANDIDATE]: Term 1 election: Requested vote from peers 4ef0a417149a4e8cb260476daacd441a (127.27.122.194:43365), 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195:41173)
I20260928 16:34:56.202469 3544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59eb8ca24376470ba80cfef02b53faca" candidate_uuid: "18222ddaedc74c41b6f47face5c2b261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ef0a417149a4e8cb260476daacd441a"
I20260928 16:34:56.202538 3684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59eb8ca24376470ba80cfef02b53faca" candidate_uuid: "18222ddaedc74c41b6f47face5c2b261" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c"
I20260928 16:34:56.202617 3544 raft_consensus.cc:3037] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:56.202657 3684 raft_consensus.cc:3037] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:56.203522 3544 raft_consensus.cc:2454] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18222ddaedc74c41b6f47face5c2b261 in term 1.
I20260928 16:34:56.203612 3684 raft_consensus.cc:2454] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18222ddaedc74c41b6f47face5c2b261 in term 1.
I20260928 16:34:56.203873 3333 leader_election.cc:302] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18222ddaedc74c41b6f47face5c2b261, 4ef0a417149a4e8cb260476daacd441a; no voters:
I20260928 16:34:56.204072 3819 raft_consensus.cc:2781] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:56.204155 3819 raft_consensus.cc:686] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [term 1 LEADER]: Becoming Leader. State: Replica: 18222ddaedc74c41b6f47face5c2b261, State: Running, Role: LEADER
I20260928 16:34:56.204316 3819 consensus_queue.cc:237] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } }
I20260928 16:34:56.205978 3243 catalog_manager.cc:5495] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18222ddaedc74c41b6f47face5c2b261 (127.27.122.193). New cstate: current_term: 1 leader_uuid: "18222ddaedc74c41b6f47face5c2b261" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18222ddaedc74c41b6f47face5c2b261" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 } health_report { overall_health: UNKNOWN } } }
I20260928 16:34:56.243428 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:56.240196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.27.122.193:40065)
0928 16:34:56.240450 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:56.240455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:56.240474 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:34:56.240512 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:56.240514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:56.240553 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:56.240654 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:34:56.240926 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:56.241462 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:56.242291 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:56.242530 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:56.242688 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:56.242931 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:34:56.243011 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:34:56.243093 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:56.243219 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 16:34:56.243272 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260928 16:34:56.249718 3684 raft_consensus.cc:1261] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Refusing update from remote peer 18222ddaedc74c41b6f47face5c2b261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:56.249774 3544 raft_consensus.cc:1261] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Refusing update from remote peer 18222ddaedc74c41b6f47face5c2b261: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:34:56.250129 3819 consensus_queue.cc:1035] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e6c90c6b1fe4f209af81e4d18b9a38c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:56.250298 3789 consensus_queue.cc:1035] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ef0a417149a4e8cb260476daacd441a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:34:56.280014 3244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:56.280175 3244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59276:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:34:56.282166 3244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04dae37594d24bbc9ba551d068f4b794: Sending DeleteTablet for 3 replicas of tablet 59eb8ca24376470ba80cfef02b53faca
I20260928 16:34:56.282711 3383 tablet_service.cc:1514] Processing DeleteTablet for tablet 59eb8ca24376470ba80cfef02b53faca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:56 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:41496
I20260928 16:34:56.282756 3524 tablet_service.cc:1514] Processing DeleteTablet for tablet 59eb8ca24376470ba80cfef02b53faca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:56 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:40936
I20260928 16:34:56.282795 3665 tablet_service.cc:1514] Processing DeleteTablet for tablet 59eb8ca24376470ba80cfef02b53faca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:34:56 UTC) from {username='oryx', principal='oryx/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:47606
I20260928 16:34:56.282884 3782 tablet_replica.cc:330] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261: stopping tablet replica
I20260928 16:34:56.282948 3783 tablet_replica.cc:330] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c: stopping tablet replica
I20260928 16:34:56.282996 3782 raft_consensus.cc:2229] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:34:56.283046 3783 raft_consensus.cc:2229] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:56.283164 3783 raft_consensus.cc:2258] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:56.283165 3782 raft_consensus.cc:2258] T 59eb8ca24376470ba80cfef02b53faca P 18222ddaedc74c41b6f47face5c2b261 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:56.283331 3781 tablet_replica.cc:330] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a: stopping tablet replica
I20260928 16:34:56.283458 3781 raft_consensus.cc:2229] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:34:56.283563 3781 raft_consensus.cc:2258] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:34:56.283586 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 3317
I20260928 16:34:56.285732 3783 ts_tablet_manager.cc:1905] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:56.287741 3783 ts_tablet_manager.cc:1918] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:56.287837 3783 log.cc:1196] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/59eb8ca24376470ba80cfef02b53faca
I20260928 16:34:56.288067 3783 ts_tablet_manager.cc:1939] T 59eb8ca24376470ba80cfef02b53faca P 5e6c90c6b1fe4f209af81e4d18b9a38c: Deleting consensus metadata
I20260928 16:34:56.288694 3230 catalog_manager.cc:4841] TS 5e6c90c6b1fe4f209af81e4d18b9a38c (127.27.122.195:41173): tablet 59eb8ca24376470ba80cfef02b53faca (table test-table [id=e0904b5601be4a4fafcbaf0cf247d5f1]) successfully deleted
I20260928 16:34:56.289147 3781 ts_tablet_manager.cc:1905] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:34:56.291071 3781 ts_tablet_manager.cc:1918] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 16:34:56.291143 3781 log.cc:1196] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/59eb8ca24376470ba80cfef02b53faca
I20260928 16:34:56.291450 3781 ts_tablet_manager.cc:1939] T 59eb8ca24376470ba80cfef02b53faca P 4ef0a417149a4e8cb260476daacd441a: Deleting consensus metadata
I20260928 16:34:56.292152 3229 catalog_manager.cc:4841] TS 4ef0a417149a4e8cb260476daacd441a (127.27.122.194:43365): tablet 59eb8ca24376470ba80cfef02b53faca (table test-table [id=e0904b5601be4a4fafcbaf0cf247d5f1]) successfully deleted
W20260928 16:34:56.292639 3232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:40065 (error 108)
I20260928 16:34:56.292928 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 3458
I20260928 16:34:56.295114 3853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:56.294384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40065 (local address 127.0.0.1:41526)
0928 16:34:56.294914 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:56.295012 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":53,"threads_started":1}
W20260928 16:34:56.295348 3232 catalog_manager.cc:4584] TS 18222ddaedc74c41b6f47face5c2b261 (127.27.122.193:40065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59eb8ca24376470ba80cfef02b53faca: Network error: Client connection negotiation failed: client connection to 127.27.122.193:40065: connect: Connection refused (error 111)
I20260928 16:34:56.300681 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 3599
I20260928 16:34:56.307495 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 3213
2026-09-28T16:34:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3439 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:56 dist-test-slave-g9sh krb5kdc[3856](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:56 dist-test-slave-g9sh krb5kdc[3856](info): set up 2 sockets
Sep 28 16:34:56 dist-test-slave-g9sh krb5kdc[3856](info): commencing operation
krb5kdc: starting...
W20260928 16:34:57.356101 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:57 dist-test-slave-g9sh krb5kdc[3856](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:34:57.475226 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:43495
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:34585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:43495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:57.582644 3869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:57.582914 3869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:57.582988 3869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:57.586905 3869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:57.586982 3869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:57.587021 3869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:57.587088 3869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:57.587126 3869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:57.592151 3869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:34585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:43495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:43495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:57.593151 3869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:57.594131 3869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:57.602377 3875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:57.602377 3874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:57.602566 3877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:57.602728 3869 server_base.cc:1034] running on GCE node
I20260928 16:34:57.603299 3869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:57.604329 3869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:57.605515 3869 hybrid_clock.cc:648] HybridClock initialized: now 1790613297605502 us; error 36 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T16:34:57Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1302 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:34:57 dist-test-slave-g9sh krb5kdc[3882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:34:57 dist-test-slave-g9sh krb5kdc[3882](info): set up 2 sockets
Sep 28 16:34:57 dist-test-slave-g9sh krb5kdc[3882](info): commencing operation
krb5kdc: starting...
W20260928 16:34:58.660215 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:34:58 dist-test-slave-g9sh krb5kdc[3882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:34:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:34:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:34:58.777588 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:35239
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:35239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:58.884095 3895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:58.884375 3895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:58.884449 3895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:58.888366 3895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:58.888454 3895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:58.888494 3895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:58.888552 3895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:58.888599 3895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:58.893814 3895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:35239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:35239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:58.894801 3895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:58.895797 3895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:58.904186 3900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:58.904196 3901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:58.904284 3895 server_base.cc:1034] running on GCE node
W20260928 16:34:58.904410 3903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:58.904987 3895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:58.905954 3895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:58.907116 3895 hybrid_clock.cc:648] HybridClock initialized: now 1790613298907094 us; error 35 us; skew 500 ppm
Sep 28 16:34:58 dist-test-slave-g9sh krb5kdc[3882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613298, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:58.909878 3895 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:34:58.910745 3895 webserver.cc:460] Webserver started at http://127.27.122.254:40447/ using document root <none> and password file <none>
I20260928 16:34:58.911280 3895 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:58.911341 3895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:58.911567 3895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:58.913618 3895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f"
format_stamp: "Formatted at 2026-09-28 16:34:58 on dist-test-slave-g9sh"
server_key: "79fc6e17816919e1d272c542faf28a67"
server_key_iv: "e3a8139e78546359916698ab4f07f6b3"
server_key_version: "encryptionkey@0"
I20260928 16:34:58.914186 3895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f"
format_stamp: "Formatted at 2026-09-28 16:34:58 on dist-test-slave-g9sh"
server_key: "79fc6e17816919e1d272c542faf28a67"
server_key_iv: "e3a8139e78546359916698ab4f07f6b3"
server_key_version: "encryptionkey@0"
I20260928 16:34:58.917707 3895 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:34:58.919915 3910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:58.921142 3895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:58.921270 3895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f"
format_stamp: "Formatted at 2026-09-28 16:34:58 on dist-test-slave-g9sh"
server_key: "79fc6e17816919e1d272c542faf28a67"
server_key_iv: "e3a8139e78546359916698ab4f07f6b3"
server_key_version: "encryptionkey@0"
I20260928 16:34:58.921378 3895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:58.937352 3895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:58.940629 3895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:58.940833 3895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:58.947695 3895 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:35239
I20260928 16:34:58.947721 3963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:35239 every 8 connection(s)
I20260928 16:34:58.948582 3895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:58.951156 3964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:34:58.953159 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 3895
I20260928 16:34:58.953303 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:34:58.953617 28139 external_mini_cluster.cc:1428] Setting key 53d6443dab4333cbf858ef68d0d8a04d
I20260928 16:34:58.958382 3964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f: Bootstrap starting.
I20260928 16:34:58.960551 3964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f: Neither blocks nor log segments found. Creating new log.
I20260928 16:34:58.961344 3964 log.cc:824] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:34:58 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:58.963307 3964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f: No bootstrap required, opened a new log
I20260928 16:34:58.966681 3964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35239 } }
I20260928 16:34:58.966920 3964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:34:58.966979 3964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7d9b4ee26a4ada9625ec9da1a6ac4f, State: Initialized, Role: FOLLOWER
I20260928 16:34:58.967438 3964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35239 } }
I20260928 16:34:58.967597 3964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:58.967676 3964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:58.967799 3964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:34:58.968137 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:58.955066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.27.122.254:35239)
0928 16:34:58.955487 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:58.955501 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:58.955534 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:34:58.956250 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:58.956277 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:58.956679 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:58.957104 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:58.958050 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:58.959102 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:58.960000 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:58.960467 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:58.960638 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:58.963305 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:58.963310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:58.963323 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:58.963350 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:58.965189 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:58.965645 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:58.965657 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:58.965662 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:58.965786 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:58.966077 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:58.966094 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:58.966096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:58.966459 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:58.966699 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:58.966969 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 16:34:58.967292 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":127,"threads_started":1}
I20260928 16:34:58.968868 3964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35239 } }
I20260928 16:34:58.969244 3964 leader_election.cc:302] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d7d9b4ee26a4ada9625ec9da1a6ac4f; no voters:
I20260928 16:34:58.969573 3964 leader_election.cc:288] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:34:58.969688 3969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:34:58.969900 3969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 1 LEADER]: Becoming Leader. State: Replica: 9d7d9b4ee26a4ada9625ec9da1a6ac4f, State: Running, Role: LEADER
I20260928 16:34:58.970194 3969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35239 } }
I20260928 16:34:58.970834 3964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:34:58.971329 3969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d7d9b4ee26a4ada9625ec9da1a6ac4f. Latest consensus state: current_term: 1 leader_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35239 } } }
I20260928 16:34:58.971459 3969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:58.971848 3974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:34:58.971954 3970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35239 } } }
I20260928 16:34:58.972138 3970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [sys.catalog]: This master's current role is: LEADER
I20260928 16:34:58.974993 3974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:34:58.980118 3974 catalog_manager.cc:1347] Generated new cluster ID: 1a671bd4f05c49e0b5b18ed2b225c755
I20260928 16:34:58.980178 3974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:34:58.999470 3974 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:34:59.000427 3974 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:34:59.004611 3974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f: Generated new TSK 0
I20260928 16:34:59.005352 3974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:34:59.038216 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35239
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:59.155998 3991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:59.156226 3991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:59.156265 3991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:59.160499 3991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:59.160555 3991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:59.160617 3991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:34:59.165669 3991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:59.166589 3991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:59.167506 3991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:59.176936 3999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:59.176961 3996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:59.176942 3997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:59.177147 3991 server_base.cc:1034] running on GCE node
I20260928 16:34:59.177595 3991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:59.178195 3991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:59.179353 3991 hybrid_clock.cc:648] HybridClock initialized: now 1790613299179339 us; error 24 us; skew 500 ppm
Sep 28 16:34:59 dist-test-slave-g9sh krb5kdc[3882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:59.186380 3991 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:34:59.187240 3991 webserver.cc:460] Webserver started at http://127.27.122.193:34483/ using document root <none> and password file <none>
I20260928 16:34:59.187743 3991 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:59.187801 3991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:59.187960 3991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:59.189983 3991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "726adebceab740d1b83718041cba5c4c"
format_stamp: "Formatted at 2026-09-28 16:34:59 on dist-test-slave-g9sh"
server_key: "20776797059570555adc6991da2f0231"
server_key_iv: "323f9a1e2c6339dee24814ebddc5143b"
server_key_version: "encryptionkey@0"
I20260928 16:34:59.190485 3991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "726adebceab740d1b83718041cba5c4c"
format_stamp: "Formatted at 2026-09-28 16:34:59 on dist-test-slave-g9sh"
server_key: "20776797059570555adc6991da2f0231"
server_key_iv: "323f9a1e2c6339dee24814ebddc5143b"
server_key_version: "encryptionkey@0"
I20260928 16:34:59.193992 3991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260928 16:34:59.196235 4006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:59.197460 3991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:59.197587 3991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "726adebceab740d1b83718041cba5c4c"
format_stamp: "Formatted at 2026-09-28 16:34:59 on dist-test-slave-g9sh"
server_key: "20776797059570555adc6991da2f0231"
server_key_iv: "323f9a1e2c6339dee24814ebddc5143b"
server_key_version: "encryptionkey@0"
I20260928 16:34:59.197687 3991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:59.211246 3991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:59.214334 3991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:59.214525 3991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:59.215070 3991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:59.215855 3991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:59.215901 3991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:59.215975 3991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:59.216006 3991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:59.224668 3991 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:35221
I20260928 16:34:59.224704 4119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:35221 every 8 connection(s)
I20260928 16:34:59.225697 3991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:34:59.234511 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 3991
I20260928 16:34:59.234658 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:34:59.234975 28139 external_mini_cluster.cc:1428] Setting key 0a5d4dbd2fbf5a7f70f643bbf005281b
Sep 28 16:34:59 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:59.239374 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:59.227398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:41011 (local address 127.27.122.254:35239)
0928 16:34:59.227559 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:59.227564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:59.228430 (+ 866us) server_negotiation.cc:407] Connection header received
0928 16:34:59.229766 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:59.229769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:59.229817 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:59.229892 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:59.230963 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:59.231486 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:59.232547 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:59.232784 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:59.232861 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:59.235596 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:59.235600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:59.235604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:59.235634 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:59.237304 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:59.237798 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:59.237804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:59.237809 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:59.237906 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:59.238285 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:59.238289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:59.238290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:59.238473 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:59.238585 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:59.239055 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 16:34:59.239202 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 16:34:59.240229 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:59.227783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:41011)
0928 16:34:59.228280 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:59.228320 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:59.229566 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:59.230047 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:59.230053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:59.230366 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:59.230836 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:59.230850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:59.231603 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:59.231606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:59.232443 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:59.232451 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:59.232965 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:59.232976 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:59.233176 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:59.233758 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:59.233777 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:59.235430 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:59.237422 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:59.237426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:59.237438 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:59.237696 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:59.237998 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:59.238001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:59.238002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:59.238173 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:59.238585 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:59.238592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:59.238825 (+ 233us) client_negotiation.cc:769] Sending connection context
0928 16:34:59.239012 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 16:34:59.239342 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":114,"threads_started":1}
I20260928 16:34:59.241426 4120 heartbeater.cc:347] Connected to a master server at 127.27.122.254:35239
I20260928 16:34:59.241684 4120 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:59.242149 4120 heartbeater.cc:511] Master 127.27.122.254:35239 requested a full tablet report, sending...
I20260928 16:34:59.243829 3928 ts_manager.cc:194] Registered new tserver with Master: 726adebceab740d1b83718041cba5c4c (127.27.122.193:35221)
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:34:59.245386 3928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:41011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:34:59.261335 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35239
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:59.371218 4125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:59.371454 4125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:59.371495 4125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:59.375842 4125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:59.375897 4125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:59.375953 4125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:34:59.381021 4125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:59.381932 4125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:59.382855 4125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:59.392339 4132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:59.392355 4130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:59.392328 4134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:59.392576 4125 server_base.cc:1034] running on GCE node
I20260928 16:34:59.393117 4125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:59.393731 4125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:59.394912 4125 hybrid_clock.cc:648] HybridClock initialized: now 1790613299394881 us; error 43 us; skew 500 ppm
Sep 28 16:34:59 dist-test-slave-g9sh krb5kdc[3882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:59.397639 4125 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:34:59.398403 4125 webserver.cc:460] Webserver started at http://127.27.122.194:37287/ using document root <none> and password file <none>
I20260928 16:34:59.398958 4125 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:59.399017 4125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:59.399243 4125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:59.401355 4125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "3a5793e9b4bc4bb6b297dc2d2fa39c18"
format_stamp: "Formatted at 2026-09-28 16:34:59 on dist-test-slave-g9sh"
server_key: "1f6fecf6800934d11ab6812e0bc2099f"
server_key_iv: "0e4ea3e07738a8237c85aed15ca5c71f"
server_key_version: "encryptionkey@0"
I20260928 16:34:59.401986 4125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "3a5793e9b4bc4bb6b297dc2d2fa39c18"
format_stamp: "Formatted at 2026-09-28 16:34:59 on dist-test-slave-g9sh"
server_key: "1f6fecf6800934d11ab6812e0bc2099f"
server_key_iv: "0e4ea3e07738a8237c85aed15ca5c71f"
server_key_version: "encryptionkey@0"
I20260928 16:34:59.405687 4125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:34:59.408145 4141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:59.409379 4125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:34:59.409498 4125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "3a5793e9b4bc4bb6b297dc2d2fa39c18"
format_stamp: "Formatted at 2026-09-28 16:34:59 on dist-test-slave-g9sh"
server_key: "1f6fecf6800934d11ab6812e0bc2099f"
server_key_iv: "0e4ea3e07738a8237c85aed15ca5c71f"
server_key_version: "encryptionkey@0"
I20260928 16:34:59.409612 4125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:59.442472 4125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:59.445325 4125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:59.445521 4125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:59.446113 4125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:59.446954 4125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:59.447001 4125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:59.447068 4125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:59.447110 4125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:59.455380 4125 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:45035
I20260928 16:34:59.455433 4254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:45035 every 8 connection(s)
I20260928 16:34:59.456272 4125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:34:59.457546 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 4125
I20260928 16:34:59.457672 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:34:59.458010 28139 external_mini_cluster.cc:1428] Setting key 3545c6dcaa231efb309cab0421e823b5
Sep 28 16:34:59 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
I20260928 16:34:59.470793 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:59.458160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:46833 (local address 127.27.122.254:35239)
0928 16:34:59.458315 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:59.458320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:59.459261 (+ 941us) server_negotiation.cc:407] Connection header received
0928 16:34:59.460652 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:59.460655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:59.460731 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:59.460869 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:59.462074 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:59.462686 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:59.463853 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:59.464085 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:59.464201 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:59.466936 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:59.466939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:59.466941 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:59.466965 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:59.468532 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:59.469056 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:59.469059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:59.469060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:59.469138 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:59.469449 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:59.469451 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:59.469453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:59.469738 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:59.469817 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:59.470386 (+ 569us) server_negotiation.cc:299] Negotiation successful
0928 16:34:59.470538 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:34:59.471298 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:59.458629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:46833)
0928 16:34:59.459080 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:59.459170 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:59.460449 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:59.461092 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:59.461105 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:59.461450 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:59.461925 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:59.461940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:59.462818 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:59.462822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:59.463713 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:59.463723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:59.464236 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:59.464241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:59.464409 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:59.464990 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:59.465006 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:59.466778 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:59.468636 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:59.468641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:59.468651 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:59.468952 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:59.469222 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:59.469224 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:59.469226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:59.469360 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:59.469846 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:59.469855 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:59.470116 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 16:34:59.470276 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 16:34:59.470510 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":133,"threads_started":1}
I20260928 16:34:59.472591 4255 heartbeater.cc:347] Connected to a master server at 127.27.122.254:35239
I20260928 16:34:59.472821 4255 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:59.473351 4255 heartbeater.cc:511] Master 127.27.122.254:35239 requested a full tablet report, sending...
I20260928 16:34:59.474406 3928 ts_manager.cc:194] Registered new tserver with Master: 3a5793e9b4bc4bb6b297dc2d2fa39c18 (127.27.122.194:45035)
I20260928 16:34:59.475262 3928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:46833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:34:59.494230 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35239
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:59.602555 4260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:59.602829 4260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:59.602902 4260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:59.606954 4260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:59.607041 4260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:59.607136 4260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:34:59.612147 4260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:59.613204 4260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:59.614251 4260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:59.623780 4268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:59.623876 4260 server_base.cc:1034] running on GCE node
W20260928 16:34:59.623795 4266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:59.624018 4265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:59.624487 4260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:59.625195 4260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:59.626395 4260 hybrid_clock.cc:648] HybridClock initialized: now 1790613299626373 us; error 30 us; skew 500 ppm
Sep 28 16:34:59 dist-test-slave-g9sh krb5kdc[3882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:59.628998 4260 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:34:59.629781 4260 webserver.cc:460] Webserver started at http://127.27.122.195:45047/ using document root <none> and password file <none>
I20260928 16:34:59.630333 4260 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:59.630425 4260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:59.630651 4260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:34:59.632622 4260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "4df9a2b02aeb46fab1063a397e29ac31"
format_stamp: "Formatted at 2026-09-28 16:34:59 on dist-test-slave-g9sh"
server_key: "f174bd5f84ba6cd4e312ca9a3712a76d"
server_key_iv: "7876dac3f0b87af99d473f475d4a69c7"
server_key_version: "encryptionkey@0"
I20260928 16:34:59.633214 4260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "4df9a2b02aeb46fab1063a397e29ac31"
format_stamp: "Formatted at 2026-09-28 16:34:59 on dist-test-slave-g9sh"
server_key: "f174bd5f84ba6cd4e312ca9a3712a76d"
server_key_iv: "7876dac3f0b87af99d473f475d4a69c7"
server_key_version: "encryptionkey@0"
I20260928 16:34:59.636672 4260 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:34:59.638965 4276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:59.640110 4260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:59.640244 4260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "4df9a2b02aeb46fab1063a397e29ac31"
format_stamp: "Formatted at 2026-09-28 16:34:59 on dist-test-slave-g9sh"
server_key: "f174bd5f84ba6cd4e312ca9a3712a76d"
server_key_iv: "7876dac3f0b87af99d473f475d4a69c7"
server_key_version: "encryptionkey@0"
I20260928 16:34:59.640352 4260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:59.652802 4260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:59.655653 4260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:59.655846 4260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:59.656391 4260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:34:59.657220 4260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:34:59.657289 4260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:59.657357 4260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:34:59.657410 4260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:59.667171 4260 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:33803
I20260928 16:34:59.667194 4389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:33803 every 8 connection(s)
I20260928 16:34:59.668002 4260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:34:59.670759 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 4260
I20260928 16:34:59.670895 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:34:59.671203 28139 external_mini_cluster.cc:1428] Setting key db5e9775ae9046fec938e0b01d388d47
Sep 28 16:34:59 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:34:59.682224 3967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:59.669820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:47045 (local address 127.27.122.254:35239)
0928 16:34:59.669951 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 16:34:59.669955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:34:59.670798 (+ 843us) server_negotiation.cc:407] Connection header received
0928 16:34:59.672610 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:34:59.672614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:34:59.672673 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:34:59.672758 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:34:59.674072 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:59.674658 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:59.675862 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:34:59.676094 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:34:59.676208 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:59.678716 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:34:59.678722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:34:59.678729 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:34:59.678752 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:34:59.680101 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:59.680629 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:59.680634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:59.680635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:59.680722 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:34:59.681159 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:34:59.681164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:34:59.681165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:34:59.681344 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:34:59.681456 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:34:59.681946 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 16:34:59.682060 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:34:59.682979 4392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:34:59.670189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:47045)
0928 16:34:59.670660 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 16:34:59.670701 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:34:59.672344 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:34:59.672973 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:34:59.672982 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:34:59.673466 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:34:59.673947 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:59.673960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:59.674787 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:59.674793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:59.675730 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:34:59.675741 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:34:59.676249 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:34:59.676253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:34:59.676438 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:34:59.677051 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:34:59.677067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:34:59.678550 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:34:59.680239 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:59.680245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:59.680258 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:59.680517 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:59.680819 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:34:59.680823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:34:59.680825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:34:59.681050 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:34:59.681456 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:34:59.681463 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:34:59.681704 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 16:34:59.681887 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 16:34:59.682211 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":159,"threads_started":1}
I20260928 16:34:59.684206 4390 heartbeater.cc:347] Connected to a master server at 127.27.122.254:35239
I20260928 16:34:59.684491 4390 heartbeater.cc:464] Registering TS with master...
I20260928 16:34:59.685082 4390 heartbeater.cc:511] Master 127.27.122.254:35239 requested a full tablet report, sending...
I20260928 16:34:59.686057 3928 ts_manager.cc:194] Registered new tserver with Master: 4df9a2b02aeb46fab1063a397e29ac31 (127.27.122.195:33803)
I20260928 16:34:59.686592 3928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:47045
I20260928 16:34:59.696722 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab.
Entry for principal oryx/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab.
WARNING: no policy specified for oryx/127.27.122.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.122.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab.
Entry for principal oryx/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab.
WARNING: no policy specified for oryx/127.27.122.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.122.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab.
Entry for principal oryx/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab.
I20260928 16:34:59.765609 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 3991
I20260928 16:34:59.771863 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 4125
I20260928 16:34:59.777287 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 4260
I20260928 16:34:59.782758 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 3895
I20260928 16:34:59.789143 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:35239
--webserver_interface=127.27.122.254
--webserver_port=40447
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:35239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:34:59.896728 4399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:34:59.897020 4399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:34:59.897109 4399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:34:59.901043 4399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:34:59.901131 4399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:34:59.901170 4399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:34:59.901206 4399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:34:59.901244 4399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:34:59.906323 4399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:35239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:35239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=40447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:34:59.907315 4399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:34:59.908310 4399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:34:59.916599 4404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:59.916602 4405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:34:59.916627 4407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:34:59.916775 4399 server_base.cc:1034] running on GCE node
I20260928 16:34:59.917393 4399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:34:59.918423 4399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:34:59.919625 4399 hybrid_clock.cc:648] HybridClock initialized: now 1790613299919594 us; error 42 us; skew 500 ppm
Sep 28 16:34:59 dist-test-slave-g9sh krb5kdc[3882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613299, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:34:59.922290 4399 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:34:59.923053 4399 webserver.cc:460] Webserver started at http://127.27.122.254:40447/ using document root <none> and password file <none>
I20260928 16:34:59.923527 4399 fs_manager.cc:359] Metadata directory not provided
I20260928 16:34:59.923602 4399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:34:59.927628 4399 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:34:59.929616 4414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:34:59.930757 4399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:34:59.930881 4399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f"
format_stamp: "Formatted at 2026-09-28 16:34:58 on dist-test-slave-g9sh"
server_key: "79fc6e17816919e1d272c542faf28a67"
server_key_iv: "e3a8139e78546359916698ab4f07f6b3"
server_key_version: "encryptionkey@0"
I20260928 16:34:59.931377 4399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:34:59.950692 4399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:34:59.953423 4399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:34:59.953605 4399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:34:59.960596 4399 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:35239
I20260928 16:34:59.960688 4467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:35239 every 8 connection(s)
I20260928 16:34:59.961668 4399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:34:59.964610 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 4399
I20260928 16:34:59.965098 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:35221
--local_ip_for_outbound_sockets=127.27.122.193
--tserver_master_addrs=127.27.122.254:35239
--webserver_port=34483
--webserver_interface=127.27.122.193
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.122.193
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:34:59.966014 4468 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:34:59.967573 4468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f: Bootstrap starting.
I20260928 16:34:59.984071 4468 log.cc:824] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:34:59.990437 4468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:34:59.990810 4468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f: Bootstrap complete.
I20260928 16:34:59.995018 4468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35239 } }
I20260928 16:34:59.995430 4468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7d9b4ee26a4ada9625ec9da1a6ac4f, State: Initialized, Role: FOLLOWER
I20260928 16:34:59.995962 4468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35239 } }
I20260928 16:34:59.996105 4468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:34:59.996197 4468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:34:59.996315 4468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:34:59.998248 4468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35239 } }
I20260928 16:34:59.998579 4468 leader_election.cc:302] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d7d9b4ee26a4ada9625ec9da1a6ac4f; no voters:
I20260928 16:34:59.998919 4468 leader_election.cc:288] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:34:59.999014 4472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:34:59.999280 4472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [term 2 LEADER]: Becoming Leader. State: Replica: 9d7d9b4ee26a4ada9625ec9da1a6ac4f, State: Running, Role: LEADER
I20260928 16:34:59.999569 4472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35239 } }
I20260928 16:35:00.000212 4468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:35:00.000546 4474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d7d9b4ee26a4ada9625ec9da1a6ac4f. Latest consensus state: current_term: 2 leader_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35239 } } }
I20260928 16:35:00.000687 4474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [sys.catalog]: This master's current role is: LEADER
I20260928 16:35:00.000890 4473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d7d9b4ee26a4ada9625ec9da1a6ac4f" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35239 } } }
I20260928 16:35:00.001071 4473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f [sys.catalog]: This master's current role is: LEADER
I20260928 16:35:00.001425 4476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:35:00.004752 4476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:35:00.005477 4476 catalog_manager.cc:1259] Loaded cluster ID: 1a671bd4f05c49e0b5b18ed2b225c755
I20260928 16:35:00.005543 4476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:35:00.007889 4476 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:35:00.008577 4476 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d7d9b4ee26a4ada9625ec9da1a6ac4f: Loaded TSK: 0
I20260928 16:35:00.009800 4476 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:35:00.076825 4470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:00.077155 4470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:00.077234 4470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:00.081310 4470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:00.081383 4470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:00.081467 4470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:35:00.086611 4470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.193.keytab
--principal=oryx/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:35221
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=34483
--tserver_master_addrs=127.27.122.254:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:00.087608 4470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:00.088573 4470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:00.099134 4497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:00.099146 4495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:00.099148 4494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:00.099808 4470 server_base.cc:1034] running on GCE node
I20260928 16:35:00.100508 4470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:00.101270 4470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:00.102491 4470 hybrid_clock.cc:648] HybridClock initialized: now 1790613300102459 us; error 46 us; skew 500 ppm
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613300, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:35:00.105194 4470 init.cc:377] Logged in from keytab as oryx/127.27.122.193@KRBTEST.COM (short username oryx)
I20260928 16:35:00.106149 4470 webserver.cc:460] Webserver started at http://127.27.122.193:34483/ using document root <none> and password file <none>
I20260928 16:35:00.106762 4470 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:00.106837 4470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:00.110836 4470 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:35:00.112874 4504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:00.114130 4470 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:35:00.114231 4470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "726adebceab740d1b83718041cba5c4c"
format_stamp: "Formatted at 2026-09-28 16:34:59 on dist-test-slave-g9sh"
server_key: "20776797059570555adc6991da2f0231"
server_key_iv: "323f9a1e2c6339dee24814ebddc5143b"
server_key_version: "encryptionkey@0"
I20260928 16:35:00.114781 4470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:00.136391 4470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:00.139372 4470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:00.139571 4470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:00.140218 4470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:35:00.141213 4470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:35:00.141288 4470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:00.141352 4470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:35:00.141384 4470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:00.150187 4470 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:35221
I20260928 16:35:00.150220 4618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:35221 every 8 connection(s)
I20260928 16:35:00.151149 4470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:35:00.161405 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 4470
I20260928 16:35:00.161841 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:45035
--local_ip_for_outbound_sockets=127.27.122.194
--tserver_master_addrs=127.27.122.254:35239
--webserver_port=37287
--webserver_interface=127.27.122.194
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.122.194
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:35:00.165302 4621 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:00.153218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:53033 (local address 127.27.122.254:35239)
0928 16:35:00.153727 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:00.153748 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:00.153974 (+ 226us) server_negotiation.cc:407] Connection header received
0928 16:35:00.155685 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:00.155697 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:00.155986 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:00.156314 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:00.157499 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.158590 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.160038 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.160508 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.160676 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.164605 (+ 3929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:53033: BlockingRecv error: recv got EOF from 127.27.122.193:53033 (error 108)
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":129,"threads_started":1}
I20260928 16:35:00.165387 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:00.153318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:53033)
0928 16:35:00.153794 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:00.153850 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:00.155170 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:00.156387 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:00.156395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:00.156836 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:00.157323 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.157337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.158722 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.158726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.159899 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.159911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.160675 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.160679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.160860 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.161534 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:00.161561 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:00.164401 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":143,"threads_started":1}
W20260928 16:35:00.165827 4619 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:35239 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.122.254:35239: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:00.171391 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:00.166289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:45643)
0928 16:35:00.166411 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:00.166427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:00.166503 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:00.166764 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:00.166767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:00.166918 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:00.167019 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.167024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.167739 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.167744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.168276 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.168284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.168727 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.168731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.168828 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.169371 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:00.169382 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:00.171259 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 16:35:00.171487 4621 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:00.166369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:45643 (local address 127.27.122.254:35239)
0928 16:35:00.166486 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:00.166490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:00.166502 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:35:00.166613 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:00.166617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:00.166665 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:00.166750 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:00.167131 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.167632 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.168394 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.168618 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.168724 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.171377 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:45643: BlockingRecv error: recv got EOF from 127.27.122.193:45643 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:00.176781 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:00.172007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:55541)
0928 16:35:00.172145 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:00.172176 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:00.172245 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:00.172462 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:00.172465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:00.172610 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:00.172673 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.172678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.173444 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.173456 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.173956 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.173964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.174388 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.174391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.174479 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.174920 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:00.174930 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:00.176650 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260928 16:35:00.176883 4621 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:00.172123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:55541 (local address 127.27.122.254:35239)
0928 16:35:00.172252 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:00.172256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:00.172270 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:35:00.172311 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:00.172315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:00.172358 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:00.172440 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:00.172779 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.173339 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.174070 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.174280 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.174386 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.176764 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:55541: BlockingRecv error: recv got EOF from 127.27.122.193:55541 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 16:35:00.177127 4619 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:35:00.275460 4623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:00.275694 4623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:00.275763 4623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:00.279772 4623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:00.279820 4623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:00.279928 4623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:35:00.284915 4623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.194.keytab
--principal=oryx/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:45035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=37287
--tserver_master_addrs=127.27.122.254:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:00.285832 4623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:00.286824 4623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:00.296284 4631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:00.296293 4629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:00.296293 4628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:00.296327 4623 server_base.cc:1034] running on GCE node
I20260928 16:35:00.296964 4623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:00.297506 4623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:00.298686 4623 hybrid_clock.cc:648] HybridClock initialized: now 1790613300298672 us; error 25 us; skew 500 ppm
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613300, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:35:00.301441 4623 init.cc:377] Logged in from keytab as oryx/127.27.122.194@KRBTEST.COM (short username oryx)
I20260928 16:35:00.302258 4623 webserver.cc:460] Webserver started at http://127.27.122.194:37287/ using document root <none> and password file <none>
I20260928 16:35:00.302742 4623 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:00.302807 4623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:00.306350 4623 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:35:00.308261 4638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:00.309485 4623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:35:00.309605 4623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "3a5793e9b4bc4bb6b297dc2d2fa39c18"
format_stamp: "Formatted at 2026-09-28 16:34:59 on dist-test-slave-g9sh"
server_key: "1f6fecf6800934d11ab6812e0bc2099f"
server_key_iv: "0e4ea3e07738a8237c85aed15ca5c71f"
server_key_version: "encryptionkey@0"
I20260928 16:35:00.310104 4623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:00.335951 4623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:00.338716 4623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:00.338907 4623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:00.339488 4623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:35:00.340291 4623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:35:00.340335 4623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:00.340402 4623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:35:00.340446 4623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:00.349241 4623 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:45035
I20260928 16:35:00.349295 4753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:45035 every 8 connection(s)
I20260928 16:35:00.350234 4623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:00.359418 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 4623
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:00.360019 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:33803
--local_ip_for_outbound_sockets=127.27.122.195
--tserver_master_addrs=127.27.122.254:35239
--webserver_port=45047
--webserver_interface=127.27.122.195
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.122.195
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:35:00.361181 4621 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:00.351980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:40745 (local address 127.27.122.254:35239)
0928 16:35:00.352114 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:00.352117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:00.352958 (+ 841us) server_negotiation.cc:407] Connection header received
0928 16:35:00.354259 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:00.354262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:00.354312 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:00.354385 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:00.355469 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.355976 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.357064 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.357277 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.357351 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.361005 (+ 3654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:40745: BlockingRecv error: recv got EOF from 127.27.122.194:40745 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:35:00.361611 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:00.352356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:40745)
0928 16:35:00.352782 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:00.352824 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:00.354061 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:00.354546 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:00.354553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:00.354888 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:00.355347 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.355360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.356080 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.356082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.356964 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.356974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.357399 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.357401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.357537 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.358121 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:00.358137 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:00.360772 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
W20260928 16:35:00.362181 4754 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:35239 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.122.254:35239: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:00.368240 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:00.362590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:33861)
0928 16:35:00.362816 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:00.362831 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:00.362911 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:00.363152 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:00.363155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:00.363308 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:00.363389 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.363394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.364148 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.364153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.364820 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.364834 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.365676 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.365679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.365757 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.366234 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:00.366244 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:00.368156 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168}
W20260928 16:35:00.368469 4621 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:00.362707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:33861 (local address 127.27.122.254:35239)
0928 16:35:00.362808 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:00.362812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:00.362873 (+ 61us) server_negotiation.cc:407] Connection header received
0928 16:35:00.363002 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:00.363005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:00.363051 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:00.363135 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:00.363498 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.364042 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.365070 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.365525 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.365657 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.368339 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:33861: BlockingRecv error: recv got EOF from 127.27.122.194:33861 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:00.373394 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:00.368746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:37569)
0928 16:35:00.368834 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:00.368848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:00.368950 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:00.369184 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:00.369187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:00.369334 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:00.369397 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.369401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.370099 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.370102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.370600 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.370607 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.371048 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.371052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.371142 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.371615 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:00.371625 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:00.373268 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 16:35:00.373486 4621 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:00.368840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:37569 (local address 127.27.122.254:35239)
0928 16:35:00.368980 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:00.368987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:00.369000 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:35:00.369035 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:00.369037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:00.369075 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:00.369170 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:00.369503 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.369986 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.370713 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.370939 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.371045 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.373375 (+ 2330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:37569: BlockingRecv error: recv got EOF from 127.27.122.194:37569 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 16:35:00.373723 4754 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:35:00.470856 4757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:00.471204 4757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:00.471289 4757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:00.475196 4757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:00.475274 4757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:00.475394 4757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:35:00.480326 4757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/krb5kdc/oryx_127.27.122.195.keytab
--principal=oryx/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:33803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=45047
--tserver_master_addrs=127.27.122.254:35239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:00.481160 4757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:00.482045 4757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:00.491397 4763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:00.491506 4757 server_base.cc:1034] running on GCE node
W20260928 16:35:00.491425 4762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:00.491415 4765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:00.492046 4757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:00.492668 4757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:00.493940 4757 hybrid_clock.cc:648] HybridClock initialized: now 1790613300493891 us; error 49 us; skew 500 ppm
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613300, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:35:00.496649 4757 init.cc:377] Logged in from keytab as oryx/127.27.122.195@KRBTEST.COM (short username oryx)
I20260928 16:35:00.497527 4757 webserver.cc:460] Webserver started at http://127.27.122.195:45047/ using document root <none> and password file <none>
I20260928 16:35:00.498085 4757 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:00.498199 4757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:00.502278 4757 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:35:00.504328 4772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:00.505569 4757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:35:00.505713 4757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "4df9a2b02aeb46fab1063a397e29ac31"
format_stamp: "Formatted at 2026-09-28 16:34:59 on dist-test-slave-g9sh"
server_key: "f174bd5f84ba6cd4e312ca9a3712a76d"
server_key_iv: "7876dac3f0b87af99d473f475d4a69c7"
server_key_version: "encryptionkey@0"
I20260928 16:35:00.506286 4757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:00.539251 4757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:00.542161 4757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:00.542373 4757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:00.542965 4757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:35:00.543792 4757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:35:00.543864 4757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:00.543938 4757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:35:00.543988 4757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:00.553184 4757 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:33803
I20260928 16:35:00.553213 4885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:33803 every 8 connection(s)
I20260928 16:35:00.554152 4757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:35:00.556694 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 4757
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 16:35:00.565806 4621 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:00.555827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:37851 (local address 127.27.122.254:35239)
0928 16:35:00.555958 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:00.555962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:00.556865 (+ 903us) server_negotiation.cc:407] Connection header received
0928 16:35:00.558677 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:00.558680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:00.558722 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:00.558794 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:00.560248 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.560740 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.562024 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.562249 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.562357 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.565677 (+ 3320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:37851: BlockingRecv error: recv got EOF from 127.27.122.195:37851 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:35:00.566552 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:00.557831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.27.122.254:35239)
0928 16:35:00.558117 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:00.558121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:00.558134 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:35:00.558181 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:00.558184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:00.558233 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:00.558332 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:00.559184 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.559978 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.560958 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.561208 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.561332 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.562725 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:00.562734 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:00.562753 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:35:00.562779 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:00.564463 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:35:00.564947 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:35:00.564961 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:35:00.564966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:35:00.565087 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:35:00.565386 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:35:00.565389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:35:00.565391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:35:00.565726 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:00.565940 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:00.566218 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 16:35:00.566436 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260928 16:35:00.566629 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:00.556164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:37851)
0928 16:35:00.556688 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:00.556747 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:00.558478 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:00.558994 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:00.559004 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:00.559482 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:00.560106 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.560128 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.560879 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.560882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.561912 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.561925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.562408 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.562412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.562588 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.563122 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:00.563146 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:00.565589 (+ 2443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
W20260928 16:35:00.567385 4886 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:35239 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.122.254:35239: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:00.572697 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:00.567831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:56785)
0928 16:35:00.568322 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:00.568359 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:00.568449 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:00.568679 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:00.568681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:00.568799 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:00.568882 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.568886 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.569705 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.569708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.570223 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.570230 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.570606 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.570609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.570671 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.571080 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:00.571092 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:00.572619 (+ 1527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434}
W20260928 16:35:00.572853 4890 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:00.567931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:56785 (local address 127.27.122.254:35239)
0928 16:35:00.568030 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:00.568034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:00.568434 (+ 400us) server_negotiation.cc:407] Connection header received
0928 16:35:00.568527 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:00.568530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:00.568563 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:00.568625 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:00.569019 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.569613 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.570315 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.570510 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.570577 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.572738 (+ 2161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:56785: BlockingRecv error: recv got EOF from 127.27.122.195:56785 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:00 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:00.577370 4888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:00.573337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:42987)
0928 16:35:00.573467 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:00.573482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:00.573544 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:00.573723 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:00.573725 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:00.573836 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:00.573894 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.573898 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.574503 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.574505 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.574978 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:00.574984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.575349 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.575352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:00.575404 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.575764 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:00.575774 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:00.577294 (+ 1520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 16:35:00.577512 4890 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:00.573424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:42987 (local address 127.27.122.254:35239)
0928 16:35:00.573551 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:00.573560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:00.573571 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:35:00.573603 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:00.573605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:00.573634 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:00.573697 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:00.573982 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.574418 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.575069 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:00.575262 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:00.575331 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:00.577406 (+ 2075us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:42987: BlockingRecv error: recv got EOF from 127.27.122.195:42987 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 16:35:00.577625 4886 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 16:35:01 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:01 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:01.183277 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:01.177748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:54875)
0928 16:35:01.178056 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:01.178069 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:01.178170 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:01.178422 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:01.178424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:01.178605 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:01.178708 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:01.178713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:01.179491 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:01.179493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:01.180058 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:01.180065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:01.180608 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:01.180611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:01.180691 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:01.181265 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:01.181278 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:01.183147 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":88,"threads_started":1}
W20260928 16:35:01.183529 4892 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:01.177895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:54875 (local address 127.27.122.254:35239)
0928 16:35:01.178131 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:01.178134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:01.178147 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:35:01.178261 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:01.178263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:01.178310 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:01.178393 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:01.178836 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:01.179392 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:01.180228 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:01.180468 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:01.180594 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:01.183310 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:54875: BlockingRecv error: recv got EOF from 127.27.122.193:54875 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":59,"threads_started":1}
Sep 28 16:35:01 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:01 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:01.379515 4893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:01.374412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:51557)
0928 16:35:01.374695 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:01.374709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:01.374807 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:01.375040 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:01.375042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:01.375224 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:01.375324 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:01.375329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:01.376055 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:01.376058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:01.376628 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:01.376636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:01.377071 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:01.377074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:01.377163 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:01.377624 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:01.377635 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:01.379397 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":99,"threads_started":1}
W20260928 16:35:01.379626 4892 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:01.374456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:51557 (local address 127.27.122.254:35239)
0928 16:35:01.374611 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:01.374616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:01.374787 (+ 171us) server_negotiation.cc:407] Connection header received
0928 16:35:01.374875 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:01.374877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:01.374922 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:01.374999 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:01.375452 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:01.375967 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:01.376728 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:01.376973 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:01.377048 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:01.379532 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:51557: BlockingRecv error: recv got EOF from 127.27.122.194:51557 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 16:35:01 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:01 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:01.583374 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:01.578303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:56855)
0928 16:35:01.578590 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:01.578607 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:01.578701 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:01.578925 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:01.578927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:01.579102 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:01.579210 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:01.579215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:01.579925 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:01.579927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:01.580487 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:01.580495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:01.580974 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:01.580983 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:01.581046 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:01.581515 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:01.581527 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:01.583253 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":113,"threads_started":1}
W20260928 16:35:01.583505 4892 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:01.578343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:56855 (local address 127.27.122.254:35239)
0928 16:35:01.578492 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:01.578496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:01.578679 (+ 183us) server_negotiation.cc:407] Connection header received
0928 16:35:01.578781 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:01.578783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:01.578833 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:01.578891 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:01.579313 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:01.579835 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:01.580597 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:01.580812 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:01.580887 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:01.583384 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:56855: BlockingRecv error: recv got EOF from 127.27.122.195:56855 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 16:35:02 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:02 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:02.189647 4895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:02.184118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:56279)
0928 16:35:02.184414 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:02.184431 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:02.184558 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:02.184927 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:02.184930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:02.185119 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:02.185219 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:02.185224 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:02.185933 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:02.185936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:02.186491 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:02.186498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:02.186986 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:02.186989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:02.187052 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:02.187572 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:02.187583 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:02.189511 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
W20260928 16:35:02.189853 4896 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:02.184154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:56279 (local address 127.27.122.254:35239)
0928 16:35:02.184387 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:02.184392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:02.184504 (+ 112us) server_negotiation.cc:407] Connection header received
0928 16:35:02.184745 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:02.184747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:02.184794 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:02.184868 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:02.185318 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:02.185842 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:02.186621 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:02.186870 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:02.186968 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:02.189671 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:56279: BlockingRecv error: recv got EOF from 127.27.122.193:56279 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":79,"threads_started":1}
Sep 28 16:35:02 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:02 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:02.385982 4897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:02.380448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:35425)
0928 16:35:02.380757 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:02.380770 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:02.380939 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:02.381216 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:02.381219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:02.381426 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:02.381523 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:02.381529 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:02.382309 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:02.382314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:02.382888 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:02.382896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:02.383368 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:02.383372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:02.383464 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:02.383956 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:02.383967 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:02.385845 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
W20260928 16:35:02.386137 4896 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:02.380508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:35425 (local address 127.27.122.254:35239)
0928 16:35:02.380695 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:02.380701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:02.380849 (+ 148us) server_negotiation.cc:407] Connection header received
0928 16:35:02.381045 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:02.381048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:02.381105 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:02.381193 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:02.381650 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:02.382195 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:02.383016 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:02.383253 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:02.383366 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:02.385982 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:35425: BlockingRecv error: recv got EOF from 127.27.122.194:35425 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 16:35:02 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:02 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:02.589848 4899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:02.584292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:40989)
0928 16:35:02.584592 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:02.584618 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:02.584717 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:02.584952 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:02.584954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:02.585227 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:02.585334 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:02.585340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:02.586211 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:02.586216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:02.586788 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:02.586796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:02.587263 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:02.587266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:02.587358 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:02.587853 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:02.587865 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:02.589648 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":121,"threads_started":1}
W20260928 16:35:02.589871 4896 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:02.584430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:40989 (local address 127.27.122.254:35239)
0928 16:35:02.584574 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:02.584578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:02.584688 (+ 110us) server_negotiation.cc:407] Connection header received
0928 16:35:02.584796 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:02.584799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:02.584846 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:02.584933 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:02.585464 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:02.586090 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:02.586913 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:02.587147 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:02.587258 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:02.589769 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:40989: BlockingRecv error: recv got EOF from 127.27.122.195:40989 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 16:35:03 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:03 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:03.195856 4900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:03.190580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:60125)
0928 16:35:03.190883 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:03.190900 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:03.191027 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:03.191324 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:03.191326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:03.191514 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:03.191620 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:03.191625 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:03.192365 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:03.192367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:03.192960 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:03.192968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:03.193403 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:03.193405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:03.193466 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:03.193954 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:03.193964 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:03.195741 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":86,"threads_started":1}
W20260928 16:35:03.196065 4901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:03.190623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:60125 (local address 127.27.122.254:35239)
0928 16:35:03.190885 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:03.190891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:03.190961 (+ 70us) server_negotiation.cc:407] Connection header received
0928 16:35:03.191130 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:03.191133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:03.191188 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:03.191284 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:03.191725 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:03.192261 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:03.193067 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:03.193299 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:03.193401 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:03.195877 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:60125: BlockingRecv error: recv got EOF from 127.27.122.193:60125 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":90,"threads_started":1}
Sep 28 16:35:03 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:03 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:03.391937 4902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:03.386745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:42969)
0928 16:35:03.387046 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:03.387060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:03.387159 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:03.387350 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:03.387352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:03.387541 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:03.387639 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:03.387645 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:03.388363 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:03.388367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:03.389120 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:03.389130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:03.389553 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:03.389556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:03.389620 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:03.390085 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:03.390096 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:03.391819 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
W20260928 16:35:03.392089 4901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:03.386811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:42969 (local address 127.27.122.254:35239)
0928 16:35:03.386968 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:03.386975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:03.387132 (+ 157us) server_negotiation.cc:407] Connection header received
0928 16:35:03.387227 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:03.387229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:03.387274 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:03.387338 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:03.387736 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:03.388259 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:03.389225 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:03.389454 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:03.389528 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:03.391966 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:42969: BlockingRecv error: recv got EOF from 127.27.122.194:42969 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 16:35:03 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:03 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:03.595927 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:03.590626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:36881)
0928 16:35:03.590913 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:03.590931 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:03.591035 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:03.591286 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:03.591289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:03.591484 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:03.591588 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:03.591595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:03.592327 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:03.592331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:03.592972 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:03.592992 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:03.593399 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:03.593402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:03.593469 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:03.594046 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:03.594059 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:03.595790 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
W20260928 16:35:03.596015 4901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:03.590734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:36881 (local address 127.27.122.254:35239)
0928 16:35:03.590879 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:03.590883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:03.591001 (+ 118us) server_negotiation.cc:407] Connection header received
0928 16:35:03.591117 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:03.591119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:03.591169 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:03.591244 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:03.591691 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:03.592208 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:03.593090 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:03.593302 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:03.593375 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:03.595922 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:36881: BlockingRecv error: recv got EOF from 127.27.122.195:36881 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 16:35:04 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:04 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:04.206133 4904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:04.196596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:33155)
0928 16:35:04.196868 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:04.196882 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:04.197021 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:04.197246 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:04.197248 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:04.197428 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:04.197529 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:04.197534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:04.198255 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:04.198257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:04.198814 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:04.198821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:04.199230 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:04.199233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:04.199294 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:04.203909 (+ 4615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:04.203921 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:04.205956 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":88,"threads_started":1}
W20260928 16:35:04.206236 4905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:04.196665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:33155 (local address 127.27.122.254:35239)
0928 16:35:04.196987 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:04.196991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:04.197003 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:35:04.197092 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:04.197094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:04.197137 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:04.197210 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:04.197637 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:04.198166 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:04.198910 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:04.199136 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:04.199210 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:04.206091 (+ 6881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:33155: BlockingRecv error: recv got EOF from 127.27.122.193:33155 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
Sep 28 16:35:04 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:04 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:04.397773 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:04.392670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:45037)
0928 16:35:04.392973 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:04.392986 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:04.393093 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:04.393327 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:04.393329 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:04.393500 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:04.393603 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:04.393609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:04.394315 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:04.394317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:04.394855 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:04.394862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:04.395268 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:04.395271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:04.395333 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:04.395796 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:04.395808 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:04.397658 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":69,"threads_started":1}
W20260928 16:35:04.397907 4905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:04.392753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:45037 (local address 127.27.122.254:35239)
0928 16:35:04.392966 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:04.392970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:04.393037 (+ 67us) server_negotiation.cc:407] Connection header received
0928 16:35:04.393183 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:04.393185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:04.393230 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:04.393297 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:04.393698 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:04.394222 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:04.394951 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:04.395172 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:04.395248 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:04.397791 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:45037: BlockingRecv error: recv got EOF from 127.27.122.194:45037 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 16:35:04 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:04 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:04.602303 4907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:04.596856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:58091)
0928 16:35:04.597203 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:04.597215 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:04.597316 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:04.597628 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:04.597631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:04.597820 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:04.597923 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:04.597928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:04.598642 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:04.598644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:04.599201 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:04.599208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:04.599614 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:04.599616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:04.599681 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:04.600142 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:04.600153 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:04.602171 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":80,"threads_started":1}
W20260928 16:35:04.602499 4905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:04.596891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:58091 (local address 127.27.122.254:35239)
0928 16:35:04.597048 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:04.597053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:04.597271 (+ 218us) server_negotiation.cc:407] Connection header received
0928 16:35:04.597419 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:04.597421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:04.597485 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:04.597570 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:04.598021 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:04.598544 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:04.599300 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:04.599512 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:04.599586 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:04.602362 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:58091: BlockingRecv error: recv got EOF from 127.27.122.195:58091 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 16:35:05 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:05 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:05.212450 4908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:05.207206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:55289)
0928 16:35:05.207480 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:05.207493 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:05.207593 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:05.207835 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:05.207837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:05.208020 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:05.208120 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:05.208126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:05.208847 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:05.208850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:05.209415 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:05.209422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:05.209889 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:05.209892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:05.209955 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:05.210442 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:05.210453 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:05.212333 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
W20260928 16:35:05.212594 4909 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:05.207252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:55289 (local address 127.27.122.254:35239)
0928 16:35:05.207491 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:05.207494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:05.207562 (+ 68us) server_negotiation.cc:407] Connection header received
0928 16:35:05.207674 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:05.207676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:05.207719 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:05.207796 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:05.208219 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:05.208757 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:05.209511 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:05.209790 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:05.209897 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:05.212462 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:55289: BlockingRecv error: recv got EOF from 127.27.122.193:55289 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
Sep 28 16:35:05 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:05 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:05.403993 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:05.398732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:40223)
0928 16:35:05.399031 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:05.399044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:05.399151 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:05.399380 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:05.399382 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:05.399564 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:05.399673 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:05.399678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:05.400430 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:05.400433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:05.401074 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:05.401084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:05.401542 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:05.401545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:05.401644 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:05.402147 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:05.402158 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:05.403877 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":105,"threads_started":1}
W20260928 16:35:05.404261 4909 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:05.398857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:40223 (local address 127.27.122.254:35239)
0928 16:35:05.399044 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:05.399048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:05.399107 (+ 59us) server_negotiation.cc:407] Connection header received
0928 16:35:05.399233 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:05.399235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:05.399280 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:05.399346 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:05.399807 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:05.400335 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:05.401217 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:05.401438 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:05.401526 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:05.404013 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:40223: BlockingRecv error: recv got EOF from 127.27.122.194:40223 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 16:35:05 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:05 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:05.608633 4911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:05.603140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:35625)
0928 16:35:05.603392 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:05.603406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:05.603499 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:05.603723 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:05.603726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:05.603912 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:05.604010 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:05.604015 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:05.604723 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:05.604725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:05.605339 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:05.605347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:05.605752 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:05.605754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:05.605818 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:05.606299 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:05.606310 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:05.608458 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
W20260928 16:35:05.608703 4909 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:05.603201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:35625 (local address 127.27.122.254:35239)
0928 16:35:05.603348 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:05.603353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:05.603460 (+ 107us) server_negotiation.cc:407] Connection header received
0928 16:35:05.603565 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:05.603568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:05.603623 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:05.603687 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:05.604109 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:05.604630 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:05.605435 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:05.605655 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:05.605729 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:05.608579 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:35625: BlockingRecv error: recv got EOF from 127.27.122.195:35625 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 16:35:06 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:06 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:06.218772 4912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:06.213346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:47025)
0928 16:35:06.213606 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:06.213621 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:06.213721 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:06.213971 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:06.213973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:06.214167 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:06.214323 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:06.214336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:06.215086 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:06.215089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:06.215659 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:06.215666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:06.216075 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:06.216078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:06.216157 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:06.216655 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:06.216666 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:06.218580 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
W20260928 16:35:06.218828 4913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:06.213400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:47025 (local address 127.27.122.254:35239)
0928 16:35:06.213637 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:06.213641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:06.213671 (+ 30us) server_negotiation.cc:407] Connection header received
0928 16:35:06.213798 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:06.213800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:06.213859 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:06.213932 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:06.214450 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:06.214983 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:06.215759 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:06.215980 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:06.216055 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:06.218706 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:47025: BlockingRecv error: recv got EOF from 127.27.122.193:47025 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":82,"threads_started":1}
Sep 28 16:35:06 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:06 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:06.410470 4914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:06.404940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:42629)
0928 16:35:06.405253 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:06.405267 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:06.405411 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:06.405719 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:06.405723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:06.405956 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:06.406057 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:06.406063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:06.406872 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:06.406874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:06.407420 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:06.407427 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:06.407838 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:06.407841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:06.407904 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:06.408424 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:06.408436 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:06.410296 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":81,"threads_started":1}
W20260928 16:35:06.410523 4913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:06.404948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:42629 (local address 127.27.122.254:35239)
0928 16:35:06.405126 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:06.405131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:06.405367 (+ 236us) server_negotiation.cc:407] Connection header received
0928 16:35:06.405517 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:06.405520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:06.405590 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:06.405662 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:06.406163 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:06.406777 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:06.407526 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:06.407742 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:06.407817 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:06.410418 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:42629: BlockingRecv error: recv got EOF from 127.27.122.194:42629 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 16:35:06 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:06 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:06.615363 4915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:06.609445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:58379)
0928 16:35:06.609757 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:06.609778 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:06.609910 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:06.610180 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:06.610183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:06.610459 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:06.610611 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:06.610624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:06.611646 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:06.611650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:06.612321 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:06.612331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:06.612767 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:06.612770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:06.612833 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:06.613447 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:06.613459 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:06.615248 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
W20260928 16:35:06.615499 4913 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:06.609587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:58379 (local address 127.27.122.254:35239)
0928 16:35:06.609733 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:06.609738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:06.609849 (+ 111us) server_negotiation.cc:407] Connection header received
0928 16:35:06.610012 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:06.610015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:06.610069 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:06.610155 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:06.610723 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:06.611507 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:06.612434 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:06.612663 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:06.612743 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:06.615378 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:58379: BlockingRecv error: recv got EOF from 127.27.122.195:58379 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 16:35:07 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:07 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:07.225229 4916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:07.219591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:55719)
0928 16:35:07.219918 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:07.219934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:07.220080 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:07.220377 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:07.220380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:07.220590 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:07.220685 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:07.220690 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:07.221503 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:07.221508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:07.222097 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:07.222105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:07.222573 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:07.222577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:07.222672 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:07.223225 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:07.223237 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:07.225020 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":69,"threads_started":1}
W20260928 16:35:07.225271 4917 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:07.219645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:55719 (local address 127.27.122.254:35239)
0928 16:35:07.219919 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:07.219925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:07.220021 (+ 96us) server_negotiation.cc:407] Connection header received
0928 16:35:07.220187 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:07.220192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:07.220246 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:07.220352 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:07.220808 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:07.221394 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:07.222217 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:07.222451 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:07.222558 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:07.225146 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:55719: BlockingRecv error: recv got EOF from 127.27.122.193:55719 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
Sep 28 16:35:07 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:07 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:07.416616 4918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:07.411269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:59167)
0928 16:35:07.411586 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:07.411599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:07.411702 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:07.411936 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:07.411938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:07.412125 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:07.412222 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:07.412227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:07.412970 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:07.412972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:07.413583 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:07.413593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:07.414001 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:07.414003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:07.414066 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:07.414643 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:07.414655 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:07.416465 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":94,"threads_started":1}
W20260928 16:35:07.416682 4917 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:07.411350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:59167 (local address 127.27.122.254:35239)
0928 16:35:07.411517 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:07.411523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:07.411673 (+ 150us) server_negotiation.cc:407] Connection header received
0928 16:35:07.411784 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:07.411786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:07.411832 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:07.411898 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:07.412317 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:07.412834 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:07.413689 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:07.413906 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:07.413985 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:07.416590 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:59167: BlockingRecv error: recv got EOF from 127.27.122.194:59167 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 16:35:07 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:07 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:07.621665 4919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:07.616358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:54765)
0928 16:35:07.616689 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:07.616703 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:07.616801 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:07.617077 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:07.617079 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:07.617282 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:07.617384 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:07.617390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:07.618109 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:07.618111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:07.618734 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:07.618744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:07.619165 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:07.619168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:07.619230 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:07.619723 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:07.619735 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:07.621546 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":101,"threads_started":1}
W20260928 16:35:07.621852 4917 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:07.616409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:54765 (local address 127.27.122.254:35239)
0928 16:35:07.616559 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:07.616563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:07.616757 (+ 194us) server_negotiation.cc:407] Connection header received
0928 16:35:07.616878 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:07.616880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:07.616979 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:07.617041 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:07.617482 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:07.618006 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:07.618838 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:07.619055 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:07.619151 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:07.621688 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:54765: BlockingRecv error: recv got EOF from 127.27.122.195:54765 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 16:35:08 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:08 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:08.236644 4920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:08.226220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:38023)
0928 16:35:08.226524 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:08.226538 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:08.226631 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:08.226855 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:08.226857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:08.227033 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:08.227148 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:08.227153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:08.228329 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:08.228333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:08.229001 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:08.229011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:08.229445 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:08.229448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:08.229509 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:08.234521 (+ 5012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:08.234533 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:08.236521 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":111,"threads_started":1}
W20260928 16:35:08.236799 4921 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:08.226392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:38023 (local address 127.27.122.254:35239)
0928 16:35:08.226644 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:08.226647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:08.226661 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:35:08.226697 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:08.226699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:08.226741 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:08.226817 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:08.227274 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:08.228210 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:08.229123 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:08.229344 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:08.229429 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:08.236660 (+ 7231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:38023: BlockingRecv error: recv got EOF from 127.27.122.193:38023 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
Sep 28 16:35:08 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:08 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:08.422885 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:08.417438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:56067)
0928 16:35:08.417720 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:08.417734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:08.417843 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:08.418051 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:08.418053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:08.418231 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:08.418428 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:08.418435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:08.419299 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:08.419301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:08.419900 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:08.419909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:08.420323 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:08.420326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:08.420388 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:08.420865 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:08.420880 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:08.422772 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":105,"threads_started":1}
W20260928 16:35:08.423061 4921 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:08.417480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:56067 (local address 127.27.122.254:35239)
0928 16:35:08.417632 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:08.417637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:08.417792 (+ 155us) server_negotiation.cc:407] Connection header received
0928 16:35:08.417908 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:08.417910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:08.417953 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:08.418017 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:08.418529 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:08.419208 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:08.420004 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:08.420226 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:08.420303 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:08.422901 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:56067: BlockingRecv error: recv got EOF from 127.27.122.194:56067 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 16:35:08 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:08 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:08.627373 4923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:08.622374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:59973)
0928 16:35:08.622651 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:08.622664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:08.622763 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:08.622982 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:08.622984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:08.623162 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:08.623267 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:08.623272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:08.623970 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:08.623972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:08.624519 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:08.624526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:08.624979 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:08.624982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:08.625046 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:08.625519 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:08.625530 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:08.627258 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
W20260928 16:35:08.627508 4921 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:08.622451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:59973 (local address 127.27.122.254:35239)
0928 16:35:08.622594 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:08.622598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:08.622728 (+ 130us) server_negotiation.cc:407] Connection header received
0928 16:35:08.622829 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:08.622831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:08.622876 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:08.622940 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:08.623366 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:08.623880 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:08.624621 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:08.624831 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:08.624960 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:08.627384 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:59973: BlockingRecv error: recv got EOF from 127.27.122.195:59973 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 16:35:09 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:09 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:09.247706 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:09.237573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:41665)
0928 16:35:09.237849 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:09.237863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:09.237957 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:09.238211 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:09.238213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:09.238397 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:09.238493 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:09.238498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:09.239243 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:09.239245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:09.239785 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:09.239792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:09.240204 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:09.240206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:09.240269 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:09.245431 (+ 5162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:09.245446 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:09.247510 (+ 2064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
W20260928 16:35:09.247763 4925 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:09.237639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:41665 (local address 127.27.122.254:35239)
0928 16:35:09.237879 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:09.237884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:09.237913 (+ 29us) server_negotiation.cc:407] Connection header received
0928 16:35:09.238044 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:09.238046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:09.238092 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:09.238173 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:09.238592 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:09.239152 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:09.239883 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:09.240102 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:09.240184 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:09.247633 (+ 7449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:41665: BlockingRecv error: recv got EOF from 127.27.122.193:41665 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":61,"threads_started":1}
Sep 28 16:35:09 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:09 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:09.429595 4926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:09.423621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:44731)
0928 16:35:09.423953 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:09.423968 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:09.424118 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:09.424393 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:09.424396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:09.424620 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:09.424717 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:09.424723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:09.425559 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:09.425564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:09.426132 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:09.426140 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:09.426602 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:09.426612 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:09.426700 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:09.427187 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:09.427199 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:09.429464 (+ 2265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":91,"threads_started":1}
W20260928 16:35:09.429731 4925 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:09.423752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:44731 (local address 127.27.122.254:35239)
0928 16:35:09.423930 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:09.423935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:09.424056 (+ 121us) server_negotiation.cc:407] Connection header received
0928 16:35:09.424221 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:09.424225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:09.424279 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:09.424367 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:09.424841 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:09.425444 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:09.426256 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:09.426486 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:09.426596 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:09.429612 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:44731: BlockingRecv error: recv got EOF from 127.27.122.194:44731 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 16:35:09 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:09 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:09.633693 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:09.628322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:56323)
0928 16:35:09.628578 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:09.628598 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:09.628696 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:09.628963 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:09.628966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:09.629148 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:09.629250 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:09.629256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:09.629969 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:09.629971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:09.630522 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:09.630529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:09.630928 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:09.630931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:09.630991 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:09.631543 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:09.631556 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:09.633487 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
W20260928 16:35:09.633783 4925 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:09.628404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:56323 (local address 127.27.122.254:35239)
0928 16:35:09.628531 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:09.628534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:09.628663 (+ 129us) server_negotiation.cc:407] Connection header received
0928 16:35:09.628761 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:09.628763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:09.628810 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:09.628876 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:09.629354 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:09.629873 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:09.630623 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:09.630833 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:09.630907 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:09.633648 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:56323: BlockingRecv error: recv got EOF from 127.27.122.195:56323 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 16:35:10 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:10 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:10.254611 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:10.248668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:37375)
0928 16:35:10.249068 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:10.249082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:10.249214 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:10.249650 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:10.249653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:10.249888 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:10.250028 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:10.250035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:10.250966 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:10.250969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:10.251550 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:10.251557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:10.252002 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:10.252005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:10.252064 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:10.252571 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:10.252581 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:10.254487 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":89,"threads_started":1}
W20260928 16:35:10.254765 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:10.248719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:37375 (local address 127.27.122.254:35239)
0928 16:35:10.249138 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:10.249146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:10.249167 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:35:10.249345 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:10.249348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:10.249456 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:10.249589 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:10.250170 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:10.250856 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:10.251669 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:10.251902 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:10.251990 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:10.254627 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:37375: BlockingRecv error: recv got EOF from 127.27.122.193:37375 (error 108)
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":217,"threads_started":1}
Sep 28 16:35:10 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:10 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:10.436370 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:10.430555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:56695)
0928 16:35:10.430863 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:10.430877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:10.430973 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:10.431217 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:10.431219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:10.431404 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:10.431528 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:10.431533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:10.432543 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:10.432554 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:10.433289 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:10.433300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:10.433721 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:10.433724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:10.433787 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:10.434337 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:10.434355 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:10.436254 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":124,"threads_started":1}
W20260928 16:35:10.436509 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:10.430607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:56695 (local address 127.27.122.254:35239)
0928 16:35:10.430771 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:10.430775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:10.430950 (+ 175us) server_negotiation.cc:407] Connection header received
0928 16:35:10.431057 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:10.431059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:10.431107 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:10.431181 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:10.431655 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:10.432417 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:10.433401 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:10.433623 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:10.433703 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:10.436388 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:56695: BlockingRecv error: recv got EOF from 127.27.122.194:56695 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 16:35:10 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:10 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:10.640034 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:10.634502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:41291)
0928 16:35:10.634775 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:10.634788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:10.634911 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:10.635196 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:10.635200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:10.635420 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:10.635543 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:10.635549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:10.636321 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:10.636326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:10.636947 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:10.636956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:10.637423 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:10.637427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:10.637535 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:10.638027 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:10.638039 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:10.639828 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":102,"threads_started":1}
W20260928 16:35:10.640148 4929 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:10.634625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:41291 (local address 127.27.122.254:35239)
0928 16:35:10.634757 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:10.634760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:10.634869 (+ 109us) server_negotiation.cc:407] Connection header received
0928 16:35:10.635014 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:10.635022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:10.635069 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:10.635173 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:10.635667 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:10.636211 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:10.637071 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:10.637310 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:10.637418 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:10.640010 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:41291: BlockingRecv error: recv got EOF from 127.27.122.195:41291 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 16:35:11 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:11 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:11.260541 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:11.255339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:35441)
0928 16:35:11.255612 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:11.255625 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:11.255719 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:11.255954 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:11.255956 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:11.256129 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:11.256226 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:11.256231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:11.256982 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:11.256984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:11.257535 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:11.257543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:11.257966 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:11.257968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:11.258029 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:11.258494 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:11.258505 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:11.260355 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
W20260928 16:35:11.260599 4933 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:11.255395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:35441 (local address 127.27.122.254:35239)
0928 16:35:11.255631 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:11.255635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:11.255685 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:35:11.255797 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:11.255799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:11.255844 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:11.255921 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:11.256319 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:11.256883 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:11.257656 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:11.257880 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:11.257950 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:11.260476 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:35441: BlockingRecv error: recv got EOF from 127.27.122.193:35441 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":62,"threads_started":1}
Sep 28 16:35:11 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:11 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:11.442394 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:11.437291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:43551)
0928 16:35:11.437552 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:11.437564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:11.437662 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:11.437866 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:11.437868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:11.438045 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:11.438153 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:11.438158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:11.438859 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:11.438862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:11.439421 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:11.439429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:11.439832 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:11.439834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:11.439896 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:11.440354 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:11.440365 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:11.442189 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
W20260928 16:35:11.442406 4933 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:11.437341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:43551 (local address 127.27.122.254:35239)
0928 16:35:11.437480 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:11.437484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:11.437639 (+ 155us) server_negotiation.cc:407] Connection header received
0928 16:35:11.437739 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:11.437741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:11.437787 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:11.437854 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:11.438243 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:11.438770 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:11.439519 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:11.439739 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:11.439813 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:11.442312 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:43551: BlockingRecv error: recv got EOF from 127.27.122.194:43551 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 16:35:11 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:11 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:11.645967 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:11.640838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:53055)
0928 16:35:11.641115 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:11.641128 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:11.641225 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:11.641456 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:11.641459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:11.641641 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:11.641744 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:11.641749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:11.642444 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:11.642447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:11.643000 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:11.643007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:11.643402 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:11.643405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:11.643467 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:11.643933 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:11.643945 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:11.645782 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":102,"threads_started":1}
W20260928 16:35:11.645994 4933 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:11.640877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:53055 (local address 127.27.122.254:35239)
0928 16:35:11.641034 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:11.641037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:11.641196 (+ 159us) server_negotiation.cc:407] Connection header received
0928 16:35:11.641304 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:11.641306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:11.641352 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:11.641419 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:11.641844 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:11.642354 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:11.643101 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:11.643313 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:11.643386 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:11.645894 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:53055: BlockingRecv error: recv got EOF from 127.27.122.195:53055 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 16:35:12 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:12 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:12.267086 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:12.261614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:42839)
0928 16:35:12.261968 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:12.261982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:12.262082 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:12.262333 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:12.262335 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:12.262519 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:12.262618 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:12.262623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:12.263380 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:12.263382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:12.264097 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:12.264115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:12.264581 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:12.264585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:12.264664 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:12.265195 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:12.265206 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:12.266973 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":148,"threads_started":1}
W20260928 16:35:12.267235 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:12.261692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:42839 (local address 127.27.122.254:35239)
0928 16:35:12.261985 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:12.261989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:12.262051 (+ 62us) server_negotiation.cc:407] Connection header received
0928 16:35:12.262173 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:12.262175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:12.262219 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:12.262299 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:12.262729 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:12.263274 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:12.264222 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:12.264467 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:12.264579 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:12.267109 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:42839: BlockingRecv error: recv got EOF from 127.27.122.193:42839 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
Sep 28 16:35:12 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:12 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:12.448721 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:12.443216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:55283)
0928 16:35:12.443476 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:12.443488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:12.443585 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:12.443817 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:12.443820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:12.443993 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:12.444096 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:12.444101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:12.444810 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:12.444812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:12.445387 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:12.445396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:12.445985 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:12.445990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:12.446081 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:12.446682 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:12.446694 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:12.448545 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
W20260928 16:35:12.448758 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:12.443297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:55283 (local address 127.27.122.254:35239)
0928 16:35:12.443431 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:12.443434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:12.443549 (+ 115us) server_negotiation.cc:407] Connection header received
0928 16:35:12.443667 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:12.443669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:12.443714 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:12.443780 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:12.444187 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:12.444724 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:12.445498 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:12.445846 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:12.445967 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:12.448661 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:55283: BlockingRecv error: recv got EOF from 127.27.122.194:55283 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 16:35:12 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:12 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:12.652281 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:12.646814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:34843)
0928 16:35:12.647091 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:12.647106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:12.647204 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:12.647459 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:12.647461 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:12.647651 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:12.647749 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:12.647755 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:12.648466 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:12.648468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:12.649089 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:12.649100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:12.649515 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:12.649518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:12.649581 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:12.650149 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:12.650160 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:12.652112 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
W20260928 16:35:12.652341 4936 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:12.646915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:34843 (local address 127.27.122.254:35239)
0928 16:35:12.647093 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:12.647097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:12.647160 (+ 63us) server_negotiation.cc:407] Connection header received
0928 16:35:12.647287 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:12.647289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:12.647355 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:12.647422 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:12.647847 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:12.648379 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:12.649192 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:12.649426 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:12.649503 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:12.652235 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:34843: BlockingRecv error: recv got EOF from 127.27.122.195:34843 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 16:35:13 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:13 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:13.273217 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:13.267964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:53549)
0928 16:35:13.268253 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:13.268267 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:13.268353 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:13.268598 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:13.268600 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:13.268790 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:13.268889 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:13.268921 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:13.269646 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:13.269648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:13.270208 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:13.270215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:13.270627 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:13.270629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:13.270691 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:13.271197 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:13.271208 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:13.273083 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":109,"threads_started":1}
W20260928 16:35:13.273350 4941 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:13.268076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:53549 (local address 127.27.122.254:35239)
0928 16:35:13.268327 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:13.268331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:13.268343 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:35:13.268435 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:13.268437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:13.268483 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:13.268559 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:13.269018 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:13.269551 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:13.270301 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:13.270526 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:13.270595 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:13.273218 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:53549: BlockingRecv error: recv got EOF from 127.27.122.193:53549 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":104,"threads_started":1}
Sep 28 16:35:13 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:13 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:13.454968 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:13.449578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:54741)
0928 16:35:13.449907 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:13.449924 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:13.450055 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:13.450318 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:13.450321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:13.450532 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:13.450635 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:13.450641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:13.451395 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:13.451400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:13.451994 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:13.452002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:13.452457 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:13.452461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:13.452550 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:13.453053 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:13.453064 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:13.454844 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":138,"threads_started":1}
W20260928 16:35:13.455130 4941 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:13.449739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:54741 (local address 127.27.122.254:35239)
0928 16:35:13.449893 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:13.449898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:13.450000 (+ 102us) server_negotiation.cc:407] Connection header received
0928 16:35:13.450156 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:13.450160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:13.450212 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:13.450295 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:13.450755 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:13.451289 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:13.452121 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:13.452348 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:13.452454 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:13.454974 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:54741: BlockingRecv error: recv got EOF from 127.27.122.194:54741 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 16:35:13 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:13 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:13.658142 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:13.653029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:57771)
0928 16:35:13.653318 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:13.653331 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:13.653432 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:13.653648 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:13.653651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:13.653832 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:13.653931 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:13.653937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:13.654628 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:13.654630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:13.655205 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:13.655213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:13.655604 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:13.655607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:13.655669 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:13.656137 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:13.656148 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:13.658012 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":122,"threads_started":1}
W20260928 16:35:13.658272 4941 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:13.653051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:57771 (local address 127.27.122.254:35239)
0928 16:35:13.653208 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:13.653212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:13.653401 (+ 189us) server_negotiation.cc:407] Connection header received
0928 16:35:13.653499 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:13.653501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:13.653548 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:13.653623 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:13.654027 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:13.654541 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:13.655301 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:13.655517 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:13.655599 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:13.658152 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:57771: BlockingRecv error: recv got EOF from 127.27.122.195:57771 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 16:35:14 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:14 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:14.284264 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:14.274002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:54501)
0928 16:35:14.274272 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:14.274290 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:14.274408 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:14.274704 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:14.274707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:14.274897 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:14.275001 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:14.275006 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:14.275731 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:14.275733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:14.276330 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:14.276337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:14.276801 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:14.276804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:14.276866 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:14.282114 (+ 5248us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:14.282132 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:14.284081 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
W20260928 16:35:14.284348 4945 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:14.274126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:54501 (local address 127.27.122.254:35239)
0928 16:35:14.274451 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:14.274454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:14.274466 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:35:14.274528 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:14.274530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:14.274590 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:14.274662 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:14.275098 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:14.275637 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:14.276426 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:14.276706 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:14.276786 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:14.284217 (+ 7431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:54501: BlockingRecv error: recv got EOF from 127.27.122.193:54501 (error 108)
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":96,"threads_started":1}
Sep 28 16:35:14 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:14 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:14.460803 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:14.455695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:42565)
0928 16:35:14.455966 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:14.455979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:14.456079 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:14.456314 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:14.456317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:14.456504 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:14.456603 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:14.456608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:14.457441 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:14.457444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:14.458012 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:14.458020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:14.458415 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:14.458418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:14.458485 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:14.458961 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:14.458973 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:14.460679 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
W20260928 16:35:14.460935 4945 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:14.455738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:42565 (local address 127.27.122.254:35239)
0928 16:35:14.455879 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:14.455883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:14.456043 (+ 160us) server_negotiation.cc:407] Connection header received
0928 16:35:14.456163 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:14.456165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:14.456210 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:14.456276 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:14.456701 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:14.457349 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:14.458112 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:14.458319 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:14.458392 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:14.460805 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:42565: BlockingRecv error: recv got EOF from 127.27.122.194:42565 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 16:35:14 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:14 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.195@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:14.664274 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:14.658981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.195:46493)
0928 16:35:14.659284 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:14.659298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:14.659394 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:14.659623 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:14.659625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:14.659804 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:14.659906 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:14.659911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:14.660674 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:14.660677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:14.661256 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:14.661265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:14.661668 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:14.661670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:14.661734 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:14.662206 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:14.662218 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:14.664090 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":133,"threads_started":1}
W20260928 16:35:14.664318 4945 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:14.659036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:46493 (local address 127.27.122.254:35239)
0928 16:35:14.659198 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:14.659203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:14.659351 (+ 148us) server_negotiation.cc:407] Connection header received
0928 16:35:14.659462 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:14.659464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:14.659510 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:14.659573 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:14.660003 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:14.660583 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:14.661353 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:14.661568 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:14.661649 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:14.664214 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.195:46493: BlockingRecv error: recv got EOF from 127.27.122.195:46493 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 16:35:15 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:15 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.193@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:15.290313 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:15.285166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.193:49873)
0928 16:35:15.285444 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:15.285458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:15.285548 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:15.285782 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:15.285784 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:15.285991 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:15.286102 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:15.286108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:15.286825 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:15.286827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:15.287384 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:15.287390 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:15.287790 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:15.287793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:15.287851 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:15.288390 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:15.288402 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:15.290193 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":90,"threads_started":1}
W20260928 16:35:15.290490 4949 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:15.285222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:49873 (local address 127.27.122.254:35239)
0928 16:35:15.285452 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:15.285457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:15.285525 (+ 68us) server_negotiation.cc:407] Connection header received
0928 16:35:15.285624 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:15.285626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:15.285673 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:15.285747 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:15.286199 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:15.286740 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:15.287477 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:15.287701 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:15.287775 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:15.290317 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.193:49873: BlockingRecv error: recv got EOF from 127.27.122.193:49873 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":61,"threads_started":1}
Sep 28 16:35:15 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 16:35:15 dist-test-slave-g9sh krb5kdc[3882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.122.194@KRBTEST.COM for oryx/127.27.122.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 16:35:15.467296 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:15.461722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35239 (local address 127.27.122.194:53943)
0928 16:35:15.462031 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:15.462044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:15.462174 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:15.462446 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:15.462449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:15.462665 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:15.462762 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:15.462769 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:15.463562 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:15.463567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:15.464180 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:15.464190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:15.464666 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:15.464670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:15.464761 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:15.465336 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:15.465348 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:15.467172 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.122.254:35239: Server oryx/127.27.122.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
W20260928 16:35:15.467509 4949 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:15.461793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:53943 (local address 127.27.122.254:35239)
0928 16:35:15.461967 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:15.461972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:15.462123 (+ 151us) server_negotiation.cc:407] Connection header received
0928 16:35:15.462274 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:15.462278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:15.462327 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:15.462416 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:15.462885 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:15.463449 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:15.464304 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:15.464552 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:15.464661 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:15.467331 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.122.194:53943: BlockingRecv error: recv got EOF from 127.27.122.194:53943 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:35:15.566943 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 4470
I20260928 16:35:15.573005 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 4623
I20260928 16:35:15.578491 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 4757
I20260928 16:35:15.583784 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 4399
2026-09-28T16:35:15Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17972 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T16:35:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:35:15Z Disabled control of system clock
I20260928 16:35:15.613524 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:46363
--webserver_interface=127.27.122.254
--webserver_port=0
--builtin_ntp_servers=127.27.122.212:40401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:46363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 16:35:15.720925 4954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:15.721336 4954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:15.721407 4954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:15.726073 4954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:35:15.726140 4954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:15.726157 4954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:15.726207 4954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:35:15.726236 4954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:35:15.731016 4954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:40401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:46363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:46363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:15.732019 4954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:15.733407 4954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:15.742022 4960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:15.742069 4954 server_base.cc:1034] running on GCE node
W20260928 16:35:15.742084 4959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:15.742029 4962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:15.742834 4954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:15.743845 4954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:15.744966 4954 hybrid_clock.cc:648] HybridClock initialized: now 1790613315744953 us; error 25 us; skew 500 ppm
I20260928 16:35:15.746750 4954 webserver.cc:460] Webserver started at http://127.27.122.254:39223/ using document root <none> and password file <none>
I20260928 16:35:15.747260 4954 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:15.747320 4954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:15.747479 4954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:35:15.749485 4954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "1fe2e51f997c499b9624009deec819f8"
format_stamp: "Formatted at 2026-09-28 16:35:15 on dist-test-slave-g9sh"
server_key: "b9b18dea752534ef635c2c9edcd42950"
server_key_iv: "ee53b749654c93534fdbc1967ee261f1"
server_key_version: "encryptionkey@0"
I20260928 16:35:15.749979 4954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "1fe2e51f997c499b9624009deec819f8"
format_stamp: "Formatted at 2026-09-28 16:35:15 on dist-test-slave-g9sh"
server_key: "b9b18dea752534ef635c2c9edcd42950"
server_key_iv: "ee53b749654c93534fdbc1967ee261f1"
server_key_version: "encryptionkey@0"
I20260928 16:35:15.753351 4954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:35:15.755503 4968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:15.756633 4954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:35:15.756755 4954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "1fe2e51f997c499b9624009deec819f8"
format_stamp: "Formatted at 2026-09-28 16:35:15 on dist-test-slave-g9sh"
server_key: "b9b18dea752534ef635c2c9edcd42950"
server_key_iv: "ee53b749654c93534fdbc1967ee261f1"
server_key_version: "encryptionkey@0"
I20260928 16:35:15.756858 4954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:15.781435 4954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:15.782074 4954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:15.782258 4954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:15.788700 4954 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:46363
I20260928 16:35:15.788743 5020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:46363 every 8 connection(s)
I20260928 16:35:15.789665 4954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:35:15.792824 5021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:35:15.798094 5021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8: Bootstrap starting.
I20260928 16:35:15.798923 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 4954
I20260928 16:35:15.799045 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:35:15.799396 28139 external_mini_cluster.cc:1428] Setting key 939ba7c05f0f1ec5497606b4f6fe037a
I20260928 16:35:15.801033 5021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8: Neither blocks nor log segments found. Creating new log.
I20260928 16:35:15.801761 5021 log.cc:824] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8: Log is configured to *not* fsync() on all Append() calls
I20260928 16:35:15.803516 5021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8: No bootstrap required, opened a new log
I20260928 16:35:15.806972 5021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe2e51f997c499b9624009deec819f8" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46363 } }
I20260928 16:35:15.807200 5021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:35:15.807281 5021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe2e51f997c499b9624009deec819f8, State: Initialized, Role: FOLLOWER
I20260928 16:35:15.807742 5021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe2e51f997c499b9624009deec819f8" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46363 } }
I20260928 16:35:15.807888 5021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:35:15.807968 5021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:35:15.808081 5021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:35:15.808200 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:15.800485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.27.122.254:46363)
0928 16:35:15.801202 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:15.801217 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:15.801251 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:35:15.801874 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:15.801902 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:15.802279 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:15.802609 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:15.803013 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:15.804020 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:15.804949 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:15.805441 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:15.805624 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:15.806443 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:15.806448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:15.806460 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:35:15.806831 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:15.806884 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:35:15.806904 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:15.807011 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:15.807125 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 16:35:15.807392 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":125,"threads_started":1}
I20260928 16:35:15.809178 5021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe2e51f997c499b9624009deec819f8" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46363 } }
I20260928 16:35:15.809515 5021 leader_election.cc:302] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fe2e51f997c499b9624009deec819f8; no voters:
I20260928 16:35:15.809818 5021 leader_election.cc:288] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:35:15.809883 5026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:35:15.810137 5026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [term 1 LEADER]: Becoming Leader. State: Replica: 1fe2e51f997c499b9624009deec819f8, State: Running, Role: LEADER
I20260928 16:35:15.810766 5026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe2e51f997c499b9624009deec819f8" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46363 } }
I20260928 16:35:15.811069 5021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:35:15.812407 5027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fe2e51f997c499b9624009deec819f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe2e51f997c499b9624009deec819f8" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46363 } } }
I20260928 16:35:15.812466 5028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fe2e51f997c499b9624009deec819f8. Latest consensus state: current_term: 1 leader_uuid: "1fe2e51f997c499b9624009deec819f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe2e51f997c499b9624009deec819f8" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46363 } } }
I20260928 16:35:15.812583 5028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [sys.catalog]: This master's current role is: LEADER
I20260928 16:35:15.812523 5027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8 [sys.catalog]: This master's current role is: LEADER
I20260928 16:35:15.813031 5038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:35:15.816057 5038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:35:15.821343 5038 catalog_manager.cc:1347] Generated new cluster ID: 9a00581c55c64a44a46febf779255f3b
I20260928 16:35:15.821411 5038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:35:15.843829 5038 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:35:15.844712 5038 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:35:15.854105 5038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fe2e51f997c499b9624009deec819f8: Generated new TSK 0
I20260928 16:35:15.854748 5038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:35:15.859710 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:46363
--builtin_ntp_servers=127.27.122.212:40401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 16:35:15.978511 5046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:15.978776 5046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:15.978842 5046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:15.982880 5046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:15.982926 5046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:15.983022 5046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:35:15.987795 5046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:40401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:46363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:15.988858 5046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:15.990257 5046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:35:15.999735 5046 server_base.cc:1034] running on GCE node
W20260928 16:35:15.999724 5052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:15.999719 5051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:15.999752 5055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:16.000313 5046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:16.000864 5046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:16.002080 5046 hybrid_clock.cc:648] HybridClock initialized: now 1790613316002062 us; error 27 us; skew 500 ppm
I20260928 16:35:16.003775 5046 webserver.cc:460] Webserver started at http://127.27.122.193:40551/ using document root <none> and password file <none>
I20260928 16:35:16.004333 5046 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:16.004395 5046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:16.004645 5046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:35:16.006718 5046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "1bec6b872f914383b62482f08e6766d9"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "e55964344ffceeda3f45e116193fd91a"
server_key_iv: "1167d8d44954b0b490fab0dd5785d0bc"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.007259 5046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "1bec6b872f914383b62482f08e6766d9"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "e55964344ffceeda3f45e116193fd91a"
server_key_iv: "1167d8d44954b0b490fab0dd5785d0bc"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.010844 5046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:35:16.013533 5061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:16.014668 5046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:35:16.014787 5046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "1bec6b872f914383b62482f08e6766d9"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "e55964344ffceeda3f45e116193fd91a"
server_key_iv: "1167d8d44954b0b490fab0dd5785d0bc"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.014889 5046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:16.029417 5046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:16.030009 5046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:16.030200 5046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:16.030750 5046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:35:16.031646 5046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:35:16.031694 5046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:16.031760 5046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:35:16.031790 5046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:16.040450 5046 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:34673
I20260928 16:35:16.040474 5174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:34673 every 8 connection(s)
I20260928 16:35:16.041332 5046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:35:16.046375 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 5046
I20260928 16:35:16.046494 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:35:16.046835 28139 external_mini_cluster.cc:1428] Setting key cf734e1e65d6c4f0156fcb3c3315f330
I20260928 16:35:16.048826 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:46363
--builtin_ntp_servers=127.27.122.212:40401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 16:35:16.050971 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:16.043055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:41731 (local address 127.27.122.254:46363)
0928 16:35:16.043194 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:16.043198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:16.044201 (+ 1003us) server_negotiation.cc:407] Connection header received
0928 16:35:16.045622 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:16.045625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:16.045671 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:16.045737 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:16.046787 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:16.047369 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:16.048804 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:16.049129 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:16.049212 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:16.050486 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:16.050489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:16.050492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:35:16.050504 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:16.050526 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:35:16.050540 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:16.050617 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:16.050774 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 16:35:16.050834 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:35:16.051793 5177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:16.043456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:46363 (local address 127.27.122.193:41731)
0928 16:35:16.044083 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:16.044123 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:16.045419 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:16.045893 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:16.045900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:16.045976 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:16.046639 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:16.046663 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:16.047492 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:16.047496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:16.048682 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:16.048697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:16.049292 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:16.049300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:16.049532 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:16.050162 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:35:16.050165 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:16.050226 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:35:16.050315 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:35:16.050623 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:35:16.050634 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:35:16.050640 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:35:16.050763 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 16:35:16.051031 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":120,"threads_started":1}
I20260928 16:35:16.053066 5175 heartbeater.cc:347] Connected to a master server at 127.27.122.254:46363
I20260928 16:35:16.053289 5175 heartbeater.cc:464] Registering TS with master...
I20260928 16:35:16.053777 5175 heartbeater.cc:511] Master 127.27.122.254:46363 requested a full tablet report, sending...
I20260928 16:35:16.055471 4985 ts_manager.cc:194] Registered new tserver with Master: 1bec6b872f914383b62482f08e6766d9 (127.27.122.193:34673)
I20260928 16:35:16.057260 4985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.122.193:41731
W20260928 16:35:16.158098 5178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:16.158337 5178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:16.158372 5178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:16.162420 5178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:16.162467 5178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:16.162518 5178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:35:16.167034 5178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:40401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:46363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:16.167968 5178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:16.169248 5178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:16.178337 5184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:16.178345 5183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:16.178361 5186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:16.178385 5178 server_base.cc:1034] running on GCE node
I20260928 16:35:16.178987 5178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:16.179570 5178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:16.180748 5178 hybrid_clock.cc:648] HybridClock initialized: now 1790613316180731 us; error 26 us; skew 500 ppm
I20260928 16:35:16.182544 5178 webserver.cc:460] Webserver started at http://127.27.122.194:35543/ using document root <none> and password file <none>
I20260928 16:35:16.183101 5178 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:16.183192 5178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:16.183414 5178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:35:16.185482 5178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "b9b07ba040ef4d398bb18592612d0ea3"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "31e5fef21ffaf5487f71d75db9f7e348"
server_key_iv: "6a429bce51982dd66067a558d2a18726"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.186038 5178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "b9b07ba040ef4d398bb18592612d0ea3"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "31e5fef21ffaf5487f71d75db9f7e348"
server_key_iv: "6a429bce51982dd66067a558d2a18726"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.189530 5178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:35:16.191937 5192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:16.193080 5178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:35:16.193203 5178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "b9b07ba040ef4d398bb18592612d0ea3"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "31e5fef21ffaf5487f71d75db9f7e348"
server_key_iv: "6a429bce51982dd66067a558d2a18726"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.193324 5178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:16.207744 5178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:16.208320 5178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:16.208458 5178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:16.209002 5178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:35:16.209862 5178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:35:16.209908 5178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:16.209944 5178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:35:16.209967 5178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:16.219002 5178 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:36045
I20260928 16:35:16.219038 5306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:36045 every 8 connection(s)
I20260928 16:35:16.219875 5178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:35:16.225029 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 5178
I20260928 16:35:16.225150 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:35:16.225440 28139 external_mini_cluster.cc:1428] Setting key 1bcfd4d835d0df62555bfd7793ddc962
I20260928 16:35:16.227324 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:46363
--builtin_ntp_servers=127.27.122.212:40401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 16:35:16.229509 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:16.221541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:33181 (local address 127.27.122.254:46363)
0928 16:35:16.221672 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:16.221676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:16.222646 (+ 970us) server_negotiation.cc:407] Connection header received
0928 16:35:16.224018 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:16.224022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:16.224085 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:16.224187 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:16.225255 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:16.225849 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:16.227206 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:16.227492 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:16.227624 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:16.228895 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:16.228923 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:16.228928 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:35:16.228948 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:16.228976 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:35:16.228990 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:16.229089 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:16.229247 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 16:35:16.229321 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:35:16.230521 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:16.221879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:46363 (local address 127.27.122.194:33181)
0928 16:35:16.222500 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:16.222539 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:16.223813 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:16.224352 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:16.224360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:16.224445 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:16.225047 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:16.225070 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:16.225986 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:16.225990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:16.227071 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:16.227082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:16.227648 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:16.227653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:16.227948 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:16.228628 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:35:16.228630 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:16.228669 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:35:16.228730 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:35:16.229097 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:35:16.229106 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:35:16.229111 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:35:16.229221 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 16:35:16.229430 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260928 16:35:16.231671 5307 heartbeater.cc:347] Connected to a master server at 127.27.122.254:46363
I20260928 16:35:16.231927 5307 heartbeater.cc:464] Registering TS with master...
I20260928 16:35:16.232576 5307 heartbeater.cc:511] Master 127.27.122.254:46363 requested a full tablet report, sending...
I20260928 16:35:16.233713 4985 ts_manager.cc:194] Registered new tserver with Master: b9b07ba040ef4d398bb18592612d0ea3 (127.27.122.194:36045)
I20260928 16:35:16.234257 4985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.122.194:33181
W20260928 16:35:16.336315 5310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:16.336572 5310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:16.336647 5310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:16.340684 5310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:16.340752 5310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:16.340831 5310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:35:16.345482 5310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:40401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:46363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:16.346429 5310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:16.347697 5310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:16.357098 5316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:16.357139 5318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:16.357139 5315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:16.357173 5310 server_base.cc:1034] running on GCE node
I20260928 16:35:16.357708 5310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:16.358270 5310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:16.359462 5310 hybrid_clock.cc:648] HybridClock initialized: now 1790613316359432 us; error 41 us; skew 500 ppm
I20260928 16:35:16.361266 5310 webserver.cc:460] Webserver started at http://127.27.122.195:39253/ using document root <none> and password file <none>
I20260928 16:35:16.361806 5310 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:16.361891 5310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:16.362109 5310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:35:16.364005 5310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "a176970927934b18ad6e87faba40107c"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "3deea479b6d2b1c5ae1fd36216b958c5"
server_key_iv: "a8be8215d094877d85ca402f3d6fd06d"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.364534 5310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "a176970927934b18ad6e87faba40107c"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "3deea479b6d2b1c5ae1fd36216b958c5"
server_key_iv: "a8be8215d094877d85ca402f3d6fd06d"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.367959 5310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:35:16.370257 5324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:16.371428 5310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:35:16.371587 5310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "a176970927934b18ad6e87faba40107c"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "3deea479b6d2b1c5ae1fd36216b958c5"
server_key_iv: "a8be8215d094877d85ca402f3d6fd06d"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.371701 5310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:16.381481 5310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:16.382048 5310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:16.382222 5310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:16.382783 5310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:35:16.383685 5310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:35:16.383752 5310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:16.383817 5310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:35:16.383864 5310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:16.392310 5310 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:42237
I20260928 16:35:16.392330 5439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:42237 every 8 connection(s)
I20260928 16:35:16.393159 5310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:35:16.393728 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 5310
I20260928 16:35:16.393834 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:35:16.394119 28139 external_mini_cluster.cc:1428] Setting key 17c48e539cf89bef8435f9483c9372ef
I20260928 16:35:16.402410 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:16.395345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:40001 (local address 127.27.122.254:46363)
0928 16:35:16.395490 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:16.395494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:16.396716 (+ 1222us) server_negotiation.cc:407] Connection header received
0928 16:35:16.398194 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:16.398197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:16.398241 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:16.398309 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:16.399108 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:16.399655 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:16.400719 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:16.400987 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:16.401065 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:16.402023 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:16.402026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:16.402029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:35:16.402040 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:16.402059 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:35:16.402067 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:16.402123 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:16.402256 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 16:35:16.402303 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:35:16.403211 5442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:16.395808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:46363 (local address 127.27.122.195:40001)
0928 16:35:16.396577 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:16.396625 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:16.398011 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:16.398437 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:16.398443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:16.398523 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:16.398995 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:16.399008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:16.399761 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:16.399763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:16.400614 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:16.400623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:16.401120 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:16.401123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:16.401255 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:16.401795 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:35:16.401797 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:16.401834 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:35:16.401895 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:35:16.402141 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:35:16.402148 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:35:16.402152 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:35:16.402241 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:35:16.402444 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":162,"threads_started":1}
I20260928 16:35:16.404290 5440 heartbeater.cc:347] Connected to a master server at 127.27.122.254:46363
I20260928 16:35:16.404503 5440 heartbeater.cc:464] Registering TS with master...
I20260928 16:35:16.405021 5440 heartbeater.cc:511] Master 127.27.122.254:46363 requested a full tablet report, sending...
I20260928 16:35:16.406015 4985 ts_manager.cc:194] Registered new tserver with Master: a176970927934b18ad6e87faba40107c (127.27.122.195:42237)
I20260928 16:35:16.406561 4985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.122.195:40001
I20260928 16:35:16.408095 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:35:16.410701 5024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:16.409865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.27.122.254:46363)
0928 16:35:16.409971 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:16.409974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:16.410128 (+ 154us) server_negotiation.cc:407] Connection header received
0928 16:35:16.410221 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:16.410223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:16.410255 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:16.410317 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:16.410581 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57584: BlockingRecv error: recv got EOF from 127.0.0.1:57584 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:35:16.410853 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 5046
I20260928 16:35:16.418538 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 5178
I20260928 16:35:16.424067 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 5310
I20260928 16:35:16.429103 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 4954
2026-09-28T16:35:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (843 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T16:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:35:16Z Disabled control of system clock
I20260928 16:35:16.457381 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:41793
--webserver_interface=127.27.122.254
--webserver_port=0
--builtin_ntp_servers=127.27.122.212:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:41793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 16:35:16.563792 5451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:16.564013 5451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:16.564046 5451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:16.567965 5451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:35:16.568013 5451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:16.568028 5451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:16.568063 5451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:35:16.568077 5451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:35:16.572561 5451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:41793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:41793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:16.573521 5451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:16.574738 5451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:16.582755 5456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:16.582823 5459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:16.582749 5457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:16.583063 5451 server_base.cc:1034] running on GCE node
I20260928 16:35:16.583670 5451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:16.584659 5451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:16.585835 5451 hybrid_clock.cc:648] HybridClock initialized: now 1790613316585817 us; error 30 us; skew 500 ppm
I20260928 16:35:16.587550 5451 webserver.cc:460] Webserver started at http://127.27.122.254:41475/ using document root <none> and password file <none>
I20260928 16:35:16.588092 5451 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:16.588187 5451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:16.588406 5451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:35:16.590400 5451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "9218ae6d492647afb5a917a98bdd2ee4"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "0d097fbc52db48248994607adf03d02c"
server_key_iv: "b2bdf841d19d5282050d682b321adeed"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.590942 5451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "9218ae6d492647afb5a917a98bdd2ee4"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "0d097fbc52db48248994607adf03d02c"
server_key_iv: "b2bdf841d19d5282050d682b321adeed"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.594484 5451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 16:35:16.596781 5465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:16.597913 5451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:35:16.598045 5451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "9218ae6d492647afb5a917a98bdd2ee4"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "0d097fbc52db48248994607adf03d02c"
server_key_iv: "b2bdf841d19d5282050d682b321adeed"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.598157 5451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:16.602442 5451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:16.602990 5451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:16.603169 5451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:16.609472 5451 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:41793
I20260928 16:35:16.609490 5518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:41793 every 8 connection(s)
I20260928 16:35:16.610314 5451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:35:16.612382 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 5451
I20260928 16:35:16.612502 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:35:16.612778 28139 external_mini_cluster.cc:1428] Setting key 2723559678f1620ea3be4a50f529fa06
I20260928 16:35:16.613710 5519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:35:16.618571 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:16.614188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.27.122.254:41793)
0928 16:35:16.614841 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:16.614860 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:16.614901 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:35:16.615699 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:16.615723 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:16.616146 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:16.616501 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:16.617206 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:16.617211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:16.617226 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:35:16.617238 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:16.617298 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:35:16.617318 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:16.617459 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:16.617526 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 16:35:16.617760 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":100,"threads_started":1}
I20260928 16:35:16.620437 5519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4: Bootstrap starting.
I20260928 16:35:16.622491 5519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4: Neither blocks nor log segments found. Creating new log.
I20260928 16:35:16.623188 5519 log.cc:824] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:35:16.624974 5519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4: No bootstrap required, opened a new log
I20260928 16:35:16.628533 5519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9218ae6d492647afb5a917a98bdd2ee4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41793 } }
I20260928 16:35:16.628732 5519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:35:16.628767 5519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9218ae6d492647afb5a917a98bdd2ee4, State: Initialized, Role: FOLLOWER
I20260928 16:35:16.629168 5519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9218ae6d492647afb5a917a98bdd2ee4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41793 } }
I20260928 16:35:16.629282 5519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:35:16.629328 5519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:35:16.629396 5519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:35:16.630213 5519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9218ae6d492647afb5a917a98bdd2ee4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41793 } }
I20260928 16:35:16.630492 5519 leader_election.cc:302] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9218ae6d492647afb5a917a98bdd2ee4; no voters:
I20260928 16:35:16.630722 5519 leader_election.cc:288] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:35:16.630831 5524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:35:16.631071 5524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [term 1 LEADER]: Becoming Leader. State: Replica: 9218ae6d492647afb5a917a98bdd2ee4, State: Running, Role: LEADER
I20260928 16:35:16.631371 5524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9218ae6d492647afb5a917a98bdd2ee4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41793 } }
I20260928 16:35:16.631960 5519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:35:16.632501 5526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9218ae6d492647afb5a917a98bdd2ee4. Latest consensus state: current_term: 1 leader_uuid: "9218ae6d492647afb5a917a98bdd2ee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9218ae6d492647afb5a917a98bdd2ee4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41793 } } }
I20260928 16:35:16.632632 5526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:35:16.632849 5525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9218ae6d492647afb5a917a98bdd2ee4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9218ae6d492647afb5a917a98bdd2ee4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41793 } } }
I20260928 16:35:16.632987 5525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:35:16.633038 5529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:35:16.635720 5529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:35:16.640570 5529 catalog_manager.cc:1347] Generated new cluster ID: 139115a87f634578ac5b0a5d077e2abb
I20260928 16:35:16.640641 5529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:35:16.664986 5529 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:35:16.665264 5529 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:35:16.680159 5529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9218ae6d492647afb5a917a98bdd2ee4: Generated new TSK 0
I20260928 16:35:16.680855 5529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:35:16.688048 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41793
--builtin_ntp_servers=127.27.122.212:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 16:35:16.798012 5543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:16.798230 5543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:16.798295 5543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:16.802354 5543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:16.802400 5543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:16.802485 5543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:35:16.807144 5543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:16.808072 5543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:16.809352 5543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:16.818590 5549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:16.818650 5543 server_base.cc:1034] running on GCE node
W20260928 16:35:16.818600 5548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:16.818605 5551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:16.819255 5543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:16.819819 5543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:16.820973 5543 hybrid_clock.cc:648] HybridClock initialized: now 1790613316820959 us; error 28 us; skew 500 ppm
I20260928 16:35:16.822754 5543 webserver.cc:460] Webserver started at http://127.27.122.193:37647/ using document root <none> and password file <none>
I20260928 16:35:16.823302 5543 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:16.823390 5543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:16.823603 5543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:35:16.825583 5543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "9b823563565c4286b0bfca4267ef8880"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "09971f7af024b7d2f1d03edfc3dff38e"
server_key_iv: "e51bf09ee892e133f75d3c5e193b1ed9"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.826139 5543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "9b823563565c4286b0bfca4267ef8880"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "09971f7af024b7d2f1d03edfc3dff38e"
server_key_iv: "e51bf09ee892e133f75d3c5e193b1ed9"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.829628 5543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:35:16.831984 5557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:16.833220 5543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:35:16.833359 5543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "9b823563565c4286b0bfca4267ef8880"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "09971f7af024b7d2f1d03edfc3dff38e"
server_key_iv: "e51bf09ee892e133f75d3c5e193b1ed9"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.833464 5543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:16.837821 5543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:16.838394 5543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:16.838567 5543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:16.839079 5543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:35:16.839975 5543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:35:16.840042 5543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:16.840116 5543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:35:16.840158 5543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:16.848403 5543 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:44353
I20260928 16:35:16.848426 5671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:44353 every 8 connection(s)
I20260928 16:35:16.849261 5543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:35:16.853847 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 5543
I20260928 16:35:16.853968 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:35:16.854271 28139 external_mini_cluster.cc:1428] Setting key 23bd3550da0e9df8dbfa14f5e9f5d9a4
I20260928 16:35:16.855057 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:16.850965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:48679 (local address 127.27.122.254:41793)
0928 16:35:16.851092 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:16.851096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:16.852039 (+ 943us) server_negotiation.cc:407] Connection header received
0928 16:35:16.853293 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:16.853296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:16.853333 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:16.853404 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:16.854539 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:16.854543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:16.854545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:35:16.854558 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:16.854579 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:35:16.854589 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:16.854650 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:16.854875 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 16:35:16.854917 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 16:35:16.856128 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41793
--builtin_ntp_servers=127.27.122.212:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 16:35:16.856171 5674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:16.851309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41793 (local address 127.27.122.193:48679)
0928 16:35:16.851908 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:16.851945 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:16.853115 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:16.853526 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:16.853532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:16.853612 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:16.854201 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:35:16.854203 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:16.854273 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:35:16.854362 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:35:16.854723 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:35:16.854730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:35:16.854736 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:35:16.855269 (+ 533us) client_negotiation.cc:240] Negotiation successful
0928 16:35:16.855492 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
I20260928 16:35:16.857318 5672 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41793
I20260928 16:35:16.857563 5672 heartbeater.cc:464] Registering TS with master...
I20260928 16:35:16.857931 5672 heartbeater.cc:511] Master 127.27.122.254:41793 requested a full tablet report, sending...
I20260928 16:35:16.859362 5482 ts_manager.cc:194] Registered new tserver with Master: 9b823563565c4286b0bfca4267ef8880 (127.27.122.193:44353)
W20260928 16:35:16.965682 5675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:16.965927 5675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:16.965993 5675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:16.969971 5675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:16.970016 5675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:16.970114 5675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:35:16.974735 5675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:16.975670 5675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:16.977032 5675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:16.986017 5681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:16.986060 5683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:16.986222 5680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:16.986162 5675 server_base.cc:1034] running on GCE node
I20260928 16:35:16.986599 5675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:16.987195 5675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:16.988374 5675 hybrid_clock.cc:648] HybridClock initialized: now 1790613316988361 us; error 29 us; skew 500 ppm
I20260928 16:35:16.990200 5675 webserver.cc:460] Webserver started at http://127.27.122.194:39279/ using document root <none> and password file <none>
I20260928 16:35:16.990760 5675 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:16.990847 5675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:16.991065 5675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:35:16.993049 5675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "8999cbad8a0348d99b0764e73a2188f1"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "934b880f2ea66ae6c10c7d5e9c3ff699"
server_key_iv: "2f77bdf30b7e28d642ae2286fe211cf1"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.993594 5675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "8999cbad8a0348d99b0764e73a2188f1"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "934b880f2ea66ae6c10c7d5e9c3ff699"
server_key_iv: "2f77bdf30b7e28d642ae2286fe211cf1"
server_key_version: "encryptionkey@0"
I20260928 16:35:16.997187 5675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:35:16.999950 5689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:17.001147 5675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:35:17.001266 5675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "8999cbad8a0348d99b0764e73a2188f1"
format_stamp: "Formatted at 2026-09-28 16:35:16 on dist-test-slave-g9sh"
server_key: "934b880f2ea66ae6c10c7d5e9c3ff699"
server_key_iv: "2f77bdf30b7e28d642ae2286fe211cf1"
server_key_version: "encryptionkey@0"
I20260928 16:35:17.001372 5675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:17.005761 5675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:17.006323 5675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:17.006491 5675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:17.006999 5675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:35:17.007848 5675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:35:17.007915 5675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:17.007980 5675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:35:17.008026 5675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:17.016535 5675 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:40849
I20260928 16:35:17.016556 5803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:40849 every 8 connection(s)
I20260928 16:35:17.017393 5675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:35:17.022164 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 5675
I20260928 16:35:17.022290 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:35:17.022575 28139 external_mini_cluster.cc:1428] Setting key b961a225048c40cceb265774b615dcb3
I20260928 16:35:17.023475 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:17.019103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:36951 (local address 127.27.122.254:41793)
0928 16:35:17.019240 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:17.019244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:17.020227 (+ 983us) server_negotiation.cc:407] Connection header received
0928 16:35:17.021569 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:17.021573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:17.021618 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:17.021699 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:17.022800 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:17.022803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:17.022806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:35:17.022819 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:17.022844 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:35:17.022859 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:17.022950 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:17.023265 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 16:35:17.023313 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:35:17.024304 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:17.019472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41793 (local address 127.27.122.194:36951)
0928 16:35:17.020087 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:17.020127 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:17.021370 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:17.021850 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:17.021856 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:17.021939 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:17.022514 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:35:17.022516 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:17.022556 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:35:17.022629 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:35:17.023119 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:35:17.023128 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:35:17.023132 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:35:17.023301 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 16:35:17.023543 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260928 16:35:17.024488 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41793
--builtin_ntp_servers=127.27.122.212:41873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 16:35:17.025435 5804 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41793
I20260928 16:35:17.025662 5804 heartbeater.cc:464] Registering TS with master...
I20260928 16:35:17.026031 5804 heartbeater.cc:511] Master 127.27.122.254:41793 requested a full tablet report, sending...
I20260928 16:35:17.026997 5482 ts_manager.cc:194] Registered new tserver with Master: 8999cbad8a0348d99b0764e73a2188f1 (127.27.122.194:40849)
W20260928 16:35:17.132282 5807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:17.132629 5807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:17.132705 5807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:17.136926 5807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:17.136996 5807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:17.137082 5807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:35:17.142232 5807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:41873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:17.143244 5807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:17.144588 5807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:17.153707 5812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:17.153710 5813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:17.153852 5807 server_base.cc:1034] running on GCE node
W20260928 16:35:17.154037 5815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:17.154417 5807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:17.154951 5807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:17.156131 5807 hybrid_clock.cc:648] HybridClock initialized: now 1790613317156125 us; error 34 us; skew 500 ppm
I20260928 16:35:17.157863 5807 webserver.cc:460] Webserver started at http://127.27.122.195:44449/ using document root <none> and password file <none>
I20260928 16:35:17.158403 5807 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:17.158492 5807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:17.158715 5807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:35:17.160671 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "4aeac3af2c824b1bb27c4447d16e41b9"
format_stamp: "Formatted at 2026-09-28 16:35:17 on dist-test-slave-g9sh"
server_key: "9a728f8d032080a6f7618e24325ed9d8"
server_key_iv: "fa15de5bea27aa8d5c071df9542f75cc"
server_key_version: "encryptionkey@0"
I20260928 16:35:17.161243 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "4aeac3af2c824b1bb27c4447d16e41b9"
format_stamp: "Formatted at 2026-09-28 16:35:17 on dist-test-slave-g9sh"
server_key: "9a728f8d032080a6f7618e24325ed9d8"
server_key_iv: "fa15de5bea27aa8d5c071df9542f75cc"
server_key_version: "encryptionkey@0"
I20260928 16:35:17.164651 5807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:35:17.167048 5821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:17.168207 5807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:35:17.168339 5807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "4aeac3af2c824b1bb27c4447d16e41b9"
format_stamp: "Formatted at 2026-09-28 16:35:17 on dist-test-slave-g9sh"
server_key: "9a728f8d032080a6f7618e24325ed9d8"
server_key_iv: "fa15de5bea27aa8d5c071df9542f75cc"
server_key_version: "encryptionkey@0"
I20260928 16:35:17.168447 5807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:17.172832 5807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:17.173415 5807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:17.173602 5807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:17.174110 5807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:35:17.174968 5807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:35:17.175033 5807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:17.175097 5807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:35:17.175145 5807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:17.184316 5807 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:42869
I20260928 16:35:17.184358 5935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:42869 every 8 connection(s)
I20260928 16:35:17.185221 5807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:35:17.190596 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 5807
I20260928 16:35:17.190685 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:35:17.190954 28139 external_mini_cluster.cc:1428] Setting key b058a5a7290aaa8cdd4ba40e1874f3f2
I20260928 16:35:17.191226 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:17.186891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:39617 (local address 127.27.122.254:41793)
0928 16:35:17.187018 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:17.187021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:17.188039 (+ 1018us) server_negotiation.cc:407] Connection header received
0928 16:35:17.189457 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:17.189460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:17.189504 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:17.189598 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:17.190753 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:17.190757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:17.190759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:35:17.190772 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:17.190793 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:35:17.190802 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:17.190897 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:17.191043 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 16:35:17.191089 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:35:17.192179 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:17.187279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41793 (local address 127.27.122.195:39617)
0928 16:35:17.187900 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:17.187938 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:17.189264 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:17.189791 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:17.189798 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:17.189879 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:17.190459 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:35:17.190460 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:17.190498 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:35:17.190563 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:35:17.190893 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:35:17.190903 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:35:17.190911 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 16:35:17.191072 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 16:35:17.191293 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
I20260928 16:35:17.193233 5936 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41793
I20260928 16:35:17.193413 5936 heartbeater.cc:464] Registering TS with master...
I20260928 16:35:17.193748 5936 heartbeater.cc:511] Master 127.27.122.254:41793 requested a full tablet report, sending...
I20260928 16:35:17.194629 5481 ts_manager.cc:194] Registered new tserver with Master: 4aeac3af2c824b1bb27c4447d16e41b9 (127.27.122.195:42869)
I20260928 16:35:17.204470 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 16:35:17.207326 5522 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:35:17.206340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54248 (local address 127.27.122.254:41793)
0928 16:35:17.206530 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:17.206534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:17.206701 (+ 167us) server_negotiation.cc:407] Connection header received
0928 16:35:17.206845 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:17.206849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:17.206881 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:17.206965 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:17.207208 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54248: BlockingRecv error: recv got EOF from 127.0.0.1:54248 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 16:35:17.207453 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 5543
I20260928 16:35:17.213248 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 5675
I20260928 16:35:17.219039 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 5807
I20260928 16:35:17.224632 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 5451
2026-09-28T16:35:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (795 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:35:17 dist-test-slave-g9sh krb5kdc[5946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:35:17 dist-test-slave-g9sh krb5kdc[5946](info): set up 2 sockets
Sep 28 16:35:17 dist-test-slave-g9sh krb5kdc[5946](info): commencing operation
krb5kdc: starting...
W20260928 16:35:18.268368 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:35:18 dist-test-slave-g9sh krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:35:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:35:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:35:18.392166 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:34081
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:34081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:35:18.520761 5959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:18.521028 5959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:18.521067 5959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:18.524870 5959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:35:18.524936 5959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:18.524956 5959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:18.524972 5959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:35:18.524987 5959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:35:18.530012 5959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:34081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:34081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:18.530994 5959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:18.531947 5959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:18.540144 5967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:18.540144 5965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:18.540282 5964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:18.540328 5959 server_base.cc:1034] running on GCE node
I20260928 16:35:18.541078 5959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:18.542127 5959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:18.543316 5959 hybrid_clock.cc:648] HybridClock initialized: now 1790613318543284 us; error 46 us; skew 500 ppm
Sep 28 16:35:18 dist-test-slave-g9sh krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613318, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:35:18.546051 5959 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:35:18.546844 5959 webserver.cc:460] Webserver started at http://127.27.122.254:42885/ using document root <none> and password file <none>
I20260928 16:35:18.547362 5959 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:18.547420 5959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:18.547574 5959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:35:18.549585 5959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "2daf5c4f71ff425388cef7ad3867d268"
format_stamp: "Formatted at 2026-09-28 16:35:18 on dist-test-slave-g9sh"
server_key: "4f04798494b2ffdfbe2d9823f4d92b61"
server_key_iv: "c0912373523e8d3a5fd8877d920f1521"
server_key_version: "encryptionkey@0"
I20260928 16:35:18.550093 5959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "2daf5c4f71ff425388cef7ad3867d268"
format_stamp: "Formatted at 2026-09-28 16:35:18 on dist-test-slave-g9sh"
server_key: "4f04798494b2ffdfbe2d9823f4d92b61"
server_key_iv: "c0912373523e8d3a5fd8877d920f1521"
server_key_version: "encryptionkey@0"
I20260928 16:35:18.553618 5959 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:35:18.555815 5974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:18.557014 5959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:35:18.557143 5959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "2daf5c4f71ff425388cef7ad3867d268"
format_stamp: "Formatted at 2026-09-28 16:35:18 on dist-test-slave-g9sh"
server_key: "4f04798494b2ffdfbe2d9823f4d92b61"
server_key_iv: "c0912373523e8d3a5fd8877d920f1521"
server_key_version: "encryptionkey@0"
I20260928 16:35:18.557258 5959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:18.574437 5959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:18.577227 5959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:18.577420 5959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:18.584473 5959 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:34081
I20260928 16:35:18.584506 6027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:34081 every 8 connection(s)
I20260928 16:35:18.585435 5959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:35:18.587829 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 5959
I20260928 16:35:18.587927 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:35:18.588196 6028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:35:18.588209 28139 external_mini_cluster.cc:1428] Setting key 652e53aebe98d5f59407b209def3014b
I20260928 16:35:18.595162 6028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268: Bootstrap starting.
Sep 28 16:35:18 dist-test-slave-g9sh krb5kdc[5946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:35:18.597877 6028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268: Neither blocks nor log segments found. Creating new log.
I20260928 16:35:18.598603 6028 log.cc:824] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268: Log is configured to *not* fsync() on all Append() calls
I20260928 16:35:18.600808 6028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268: No bootstrap required, opened a new log
I20260928 16:35:18.602799 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:18.589606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54562 (local address 127.27.122.254:34081)
0928 16:35:18.589980 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:18.590002 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:18.590035 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:35:18.590829 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:18.590847 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:18.591167 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:18.591430 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:18.592361 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:18.593417 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:18.594567 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:18.595016 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:18.595134 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:18.597833 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:18.597838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:18.597851 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:35:18.597877 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:18.599876 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:35:18.600283 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:35:18.600289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:35:18.600294 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:35:18.600427 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:35:18.600687 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:35:18.600690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:35:18.600692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:35:18.601204 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:18.601404 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:18.601588 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 16:35:18.601877 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":126,"threads_started":1}
I20260928 16:35:18.604744 6028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daf5c4f71ff425388cef7ad3867d268" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34081 } }
I20260928 16:35:18.605037 6028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:35:18.605085 6028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2daf5c4f71ff425388cef7ad3867d268, State: Initialized, Role: FOLLOWER
I20260928 16:35:18.605482 6028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daf5c4f71ff425388cef7ad3867d268" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34081 } }
I20260928 16:35:18.605609 6028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:35:18.605660 6028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:35:18.605733 6028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:35:18.606495 6028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daf5c4f71ff425388cef7ad3867d268" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34081 } }
I20260928 16:35:18.606775 6028 leader_election.cc:302] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2daf5c4f71ff425388cef7ad3867d268; no voters:
I20260928 16:35:18.607013 6028 leader_election.cc:288] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:35:18.607131 6033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:35:18.607352 6033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [term 1 LEADER]: Becoming Leader. State: Replica: 2daf5c4f71ff425388cef7ad3867d268, State: Running, Role: LEADER
I20260928 16:35:18.607702 6033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daf5c4f71ff425388cef7ad3867d268" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34081 } }
I20260928 16:35:18.608220 6028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:35:18.609289 6035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2daf5c4f71ff425388cef7ad3867d268. Latest consensus state: current_term: 1 leader_uuid: "2daf5c4f71ff425388cef7ad3867d268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daf5c4f71ff425388cef7ad3867d268" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34081 } } }
I20260928 16:35:18.609407 6035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [sys.catalog]: This master's current role is: LEADER
I20260928 16:35:18.609777 6042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:35:18.609763 6034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2daf5c4f71ff425388cef7ad3867d268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daf5c4f71ff425388cef7ad3867d268" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 34081 } } }
I20260928 16:35:18.609992 6034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268 [sys.catalog]: This master's current role is: LEADER
I20260928 16:35:18.612452 6042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:35:18.617558 6042 catalog_manager.cc:1347] Generated new cluster ID: b2fb28182f03458291ca8dac752f2528
I20260928 16:35:18.617620 6042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:35:18.628959 6042 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:35:18.630112 6042 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:35:18.635612 6042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268: Generated new TSK 0
I20260928 16:35:18.636296 6042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:35:18.667984 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34081
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:35:18.777655 6054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:18.777879 6054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:18.777947 6054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:18.782522 6054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:18.782572 6054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:18.782665 6054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:35:18.787727 6054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:18.788697 6054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:18.789695 6054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:18.798913 6062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:18.798887 6060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:18.798970 6059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:18.799043 6054 server_base.cc:1034] running on GCE node
I20260928 16:35:18.799649 6054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:18.800201 6054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:18.801383 6054 hybrid_clock.cc:648] HybridClock initialized: now 1790613318801337 us; error 57 us; skew 500 ppm
Sep 28 16:35:18 dist-test-slave-g9sh krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613318, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:35:18.804085 6054 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:35:18.804893 6054 webserver.cc:460] Webserver started at http://127.27.122.193:34669/ using document root <none> and password file <none>
I20260928 16:35:18.805482 6054 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:18.805568 6054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:18.805801 6054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:35:18.807802 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "1d381b12661a48ba8aede1b2416ed823"
format_stamp: "Formatted at 2026-09-28 16:35:18 on dist-test-slave-g9sh"
server_key: "4e4294014e15221c1195873dac23a0e2"
server_key_iv: "e2f445bae7597a1925f36db0ca150b85"
server_key_version: "encryptionkey@0"
I20260928 16:35:18.808364 6054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "1d381b12661a48ba8aede1b2416ed823"
format_stamp: "Formatted at 2026-09-28 16:35:18 on dist-test-slave-g9sh"
server_key: "4e4294014e15221c1195873dac23a0e2"
server_key_iv: "e2f445bae7597a1925f36db0ca150b85"
server_key_version: "encryptionkey@0"
I20260928 16:35:18.812048 6054 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 16:35:18.814495 6069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:18.815654 6054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:35:18.815783 6054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "1d381b12661a48ba8aede1b2416ed823"
format_stamp: "Formatted at 2026-09-28 16:35:18 on dist-test-slave-g9sh"
server_key: "4e4294014e15221c1195873dac23a0e2"
server_key_iv: "e2f445bae7597a1925f36db0ca150b85"
server_key_version: "encryptionkey@0"
I20260928 16:35:18.815884 6054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:18.826543 6054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:18.829241 6054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:18.829425 6054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:18.829962 6054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:35:18.830737 6054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:35:18.830803 6054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:18.830874 6054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:35:18.830915 6054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:18.839430 6054 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:44621
I20260928 16:35:18.839474 6183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:44621 every 8 connection(s)
I20260928 16:35:18.840263 6054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:35:18.843606 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 6054
I20260928 16:35:18.843746 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:35:18.844017 28139 external_mini_cluster.cc:1428] Setting key 6468be2b643f08363bbfad1786098ac8
Sep 28 16:35:18 dist-test-slave-g9sh krb5kdc[5946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613318, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:35:18.854244 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:18.841949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:56493 (local address 127.27.122.254:34081)
0928 16:35:18.842072 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:18.842075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:18.842846 (+ 771us) server_negotiation.cc:407] Connection header received
0928 16:35:18.844362 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:18.844365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:18.844427 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:18.844520 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:18.845824 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:18.846357 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:18.847480 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:18.847732 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:18.847842 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:18.850652 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:18.850655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:18.850657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:35:18.850680 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:18.852090 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:35:18.852686 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:35:18.852688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:35:18.852689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:35:18.852761 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:35:18.853175 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:35:18.853177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:35:18.853178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:35:18.853340 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:18.853417 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:18.854029 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 16:35:18.854129 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:35:18.855019 6186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:18.842290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:34081 (local address 127.27.122.193:56493)
0928 16:35:18.842710 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:18.842751 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:18.844136 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:18.844698 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:18.844707 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:18.845215 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:18.845686 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:18.845700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:18.846484 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:18.846488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:18.847355 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:18.847365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:18.847881 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:18.847886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:18.848055 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:18.848639 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:18.848654 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:18.850478 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:35:18.852213 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:35:18.852220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:35:18.852245 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:35:18.852590 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:35:18.852850 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:35:18.852855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:35:18.852859 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:35:18.853067 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:35:18.853447 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:35:18.853455 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:35:18.853776 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 16:35:18.853965 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 16:35:18.854209 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
I20260928 16:35:18.856261 6184 heartbeater.cc:347] Connected to a master server at 127.27.122.254:34081
I20260928 16:35:18.856493 6184 heartbeater.cc:464] Registering TS with master...
I20260928 16:35:18.857002 6184 heartbeater.cc:511] Master 127.27.122.254:34081 requested a full tablet report, sending...
I20260928 16:35:18.858434 5991 ts_manager.cc:194] Registered new tserver with Master: 1d381b12661a48ba8aede1b2416ed823 (127.27.122.193:44621)
I20260928 16:35:18.859525 5991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:56493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:35:18.871781 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34081
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:35:18.978986 6189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:18.979270 6189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:18.979342 6189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:18.983317 6189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:18.983393 6189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:18.983480 6189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:35:18.988516 6189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:18.989549 6189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:18.990530 6189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:18.999967 6195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:18.999986 6196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:18.999967 6198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:19.000191 6189 server_base.cc:1034] running on GCE node
I20260928 16:35:19.000579 6189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:19.001268 6189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:19.002445 6189 hybrid_clock.cc:648] HybridClock initialized: now 1790613319002413 us; error 44 us; skew 500 ppm
Sep 28 16:35:18 dist-test-slave-g9sh krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613318, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:35:19.005033 6189 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:35:19.005882 6189 webserver.cc:460] Webserver started at http://127.27.122.194:38873/ using document root <none> and password file <none>
I20260928 16:35:19.006424 6189 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:19.006511 6189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:19.006727 6189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:35:19.008761 6189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "4fc7a86e2271467fa0a76af69137296c"
format_stamp: "Formatted at 2026-09-28 16:35:19 on dist-test-slave-g9sh"
server_key: "d1eb92449875f9e8bb0ad39ef2bc696e"
server_key_iv: "be5823e088193fd4764db4a154fbb08a"
server_key_version: "encryptionkey@0"
I20260928 16:35:19.009370 6189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "4fc7a86e2271467fa0a76af69137296c"
format_stamp: "Formatted at 2026-09-28 16:35:19 on dist-test-slave-g9sh"
server_key: "d1eb92449875f9e8bb0ad39ef2bc696e"
server_key_iv: "be5823e088193fd4764db4a154fbb08a"
server_key_version: "encryptionkey@0"
I20260928 16:35:19.012971 6189 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:35:19.015216 6205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:19.016372 6189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:35:19.016497 6189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "4fc7a86e2271467fa0a76af69137296c"
format_stamp: "Formatted at 2026-09-28 16:35:19 on dist-test-slave-g9sh"
server_key: "d1eb92449875f9e8bb0ad39ef2bc696e"
server_key_iv: "be5823e088193fd4764db4a154fbb08a"
server_key_version: "encryptionkey@0"
I20260928 16:35:19.016597 6189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:19.041538 6189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:19.044281 6189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:19.044479 6189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:19.045086 6189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:35:19.045902 6189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:35:19.045969 6189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:19.046036 6189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:35:19.046082 6189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:19.054731 6189 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:43619
I20260928 16:35:19.054759 6319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:43619 every 8 connection(s)
I20260928 16:35:19.055655 6189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:35:19.057147 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 6189
I20260928 16:35:19.057260 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:35:19.057572 28139 external_mini_cluster.cc:1428] Setting key fbc1b86eb25fd3c29120f9b4d8964344
Sep 28 16:35:19 dist-test-slave-g9sh krb5kdc[5946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613318, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
I20260928 16:35:19.070317 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.057367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:60295 (local address 127.27.122.254:34081)
0928 16:35:19.057504 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:19.057509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:19.058503 (+ 994us) server_negotiation.cc:407] Connection header received
0928 16:35:19.060454 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:19.060457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:19.060516 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:19.060605 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:19.061980 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.062584 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.063660 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.064096 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.064220 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.066913 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:19.066917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:19.066919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:35:19.066942 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:19.068446 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:35:19.068894 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:35:19.068897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:35:19.068935 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:35:19.069015 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:35:19.069336 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:35:19.069339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:35:19.069340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:35:19.069513 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:19.069591 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:19.070052 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 16:35:19.070197 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
I20260928 16:35:19.071097 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.057819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:34081 (local address 127.27.122.194:60295)
0928 16:35:19.058370 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:19.058414 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:19.060236 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:19.060805 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:19.060816 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:19.061320 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:19.061838 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.061854 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.062690 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.062693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.063536 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.063544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.064242 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.064245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.064379 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.064998 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:19.065013 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:19.066746 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:35:19.068549 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:35:19.068553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:35:19.068564 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:35:19.068811 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:35:19.069093 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:35:19.069096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:35:19.069099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:35:19.069252 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:35:19.069608 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:35:19.069613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:35:19.069820 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 16:35:19.069967 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 16:35:19.070199 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":141,"threads_started":1}
I20260928 16:35:19.072170 6320 heartbeater.cc:347] Connected to a master server at 127.27.122.254:34081
I20260928 16:35:19.072408 6320 heartbeater.cc:464] Registering TS with master...
I20260928 16:35:19.072863 6320 heartbeater.cc:511] Master 127.27.122.254:34081 requested a full tablet report, sending...
I20260928 16:35:19.073920 5991 ts_manager.cc:194] Registered new tserver with Master: 4fc7a86e2271467fa0a76af69137296c (127.27.122.194:43619)
I20260928 16:35:19.074560 5991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:60295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:35:19.085999 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34081
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:35:19.194957 6325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:35:19.195225 6325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:35:19.195294 6325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:35:19.199523 6325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:35:19.199599 6325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:35:19.199678 6325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:35:19.204797 6325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:34081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:35:19.205808 6325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:35:19.206810 6325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:35:19.216215 6330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:19.216223 6333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:35:19.216231 6331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:35:19.216286 6325 server_base.cc:1034] running on GCE node
I20260928 16:35:19.216862 6325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:35:19.217468 6325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:35:19.218658 6325 hybrid_clock.cc:648] HybridClock initialized: now 1790613319218645 us; error 33 us; skew 500 ppm
Sep 28 16:35:19 dist-test-slave-g9sh krb5kdc[5946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613319, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:35:19.221534 6325 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:35:19.222397 6325 webserver.cc:460] Webserver started at http://127.27.122.195:33159/ using document root <none> and password file <none>
I20260928 16:35:19.222888 6325 fs_manager.cc:359] Metadata directory not provided
I20260928 16:35:19.222946 6325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:35:19.223102 6325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:35:19.225049 6325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "a287305093704ecba375ea243fb77bfb"
format_stamp: "Formatted at 2026-09-28 16:35:19 on dist-test-slave-g9sh"
server_key: "31b3bc94c5c5f2991327ee5a837016ab"
server_key_iv: "f95d2a43c907d0369ee84219ed7421cd"
server_key_version: "encryptionkey@0"
I20260928 16:35:19.225553 6325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "a287305093704ecba375ea243fb77bfb"
format_stamp: "Formatted at 2026-09-28 16:35:19 on dist-test-slave-g9sh"
server_key: "31b3bc94c5c5f2991327ee5a837016ab"
server_key_iv: "f95d2a43c907d0369ee84219ed7421cd"
server_key_version: "encryptionkey@0"
I20260928 16:35:19.229063 6325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:35:19.231320 6340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:19.232496 6325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:35:19.232631 6325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "a287305093704ecba375ea243fb77bfb"
format_stamp: "Formatted at 2026-09-28 16:35:19 on dist-test-slave-g9sh"
server_key: "31b3bc94c5c5f2991327ee5a837016ab"
server_key_iv: "f95d2a43c907d0369ee84219ed7421cd"
server_key_version: "encryptionkey@0"
I20260928 16:35:19.232729 6325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:35:19.246201 6325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:35:19.249311 6325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:35:19.249498 6325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:35:19.250041 6325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:35:19.250813 6325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:35:19.250857 6325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:19.250922 6325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:35:19.250959 6325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:35:19.259374 6325 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:46843
I20260928 16:35:19.259398 6454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:46843 every 8 connection(s)
I20260928 16:35:19.260285 6325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:35:19.261436 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 6325
I20260928 16:35:19.261559 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:35:19.261879 28139 external_mini_cluster.cc:1428] Setting key 1b9996beefefd8b3390dc470a95a3c81
Sep 28 16:35:19 dist-test-slave-g9sh krb5kdc[5946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613319, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:35:19.273574 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.262201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:58799 (local address 127.27.122.254:34081)
0928 16:35:19.262348 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:19.262352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:19.263295 (+ 943us) server_negotiation.cc:407] Connection header received
0928 16:35:19.264865 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:19.264868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:19.264947 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:19.265022 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:19.266125 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.266668 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.267805 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.268046 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.268155 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.270413 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:19.270416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:19.270418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:35:19.270441 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:19.271820 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:35:19.272257 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:35:19.272260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:35:19.272261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:35:19.272332 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:35:19.272673 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:35:19.272675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:35:19.272676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:35:19.272829 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:19.272961 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:19.273369 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 16:35:19.273460 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:35:19.274350 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.262642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:34081 (local address 127.27.122.195:58799)
0928 16:35:19.263164 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:19.263204 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:19.264661 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:19.265207 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:19.265214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:19.265543 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:35:19.265987 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.266001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.266792 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.266796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.267680 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.267690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.268192 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.268197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.268361 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.268815 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:35:19.268830 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:35:19.270267 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:35:19.271913 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:35:19.271917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:35:19.271926 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:35:19.272170 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:35:19.272448 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:35:19.272450 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:35:19.272452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:35:19.272594 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:35:19.272967 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:35:19.272977 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:35:19.273177 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 16:35:19.273327 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 16:35:19.273559 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":131,"threads_started":1}
I20260928 16:35:19.275525 6455 heartbeater.cc:347] Connected to a master server at 127.27.122.254:34081
I20260928 16:35:19.275761 6455 heartbeater.cc:464] Registering TS with master...
I20260928 16:35:19.276221 6455 heartbeater.cc:511] Master 127.27.122.254:34081 requested a full tablet report, sending...
I20260928 16:35:19.277222 5991 ts_manager.cc:194] Registered new tserver with Master: a287305093704ecba375ea243fb77bfb (127.27.122.195:46843)
I20260928 16:35:19.277766 5991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:58799
I20260928 16:35:19.287051 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:35:19.299016 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.288823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.27.122.254:34081)
0928 16:35:19.288968 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:19.288971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:19.289163 (+ 192us) server_negotiation.cc:407] Connection header received
0928 16:35:19.289257 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:19.289259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:19.289295 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:19.289364 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:35:19.289962 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.290442 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.291193 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.291393 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.291466 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.296411 (+ 4945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:35:19.296414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:35:19.296416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:35:19.296438 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:35:19.297668 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:35:19.298054 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:35:19.298057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:35:19.298058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:35:19.298135 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:35:19.298349 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:35:19.298351 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:35:19.298352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:35:19.298546 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:35:19.298656 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:19.298824 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 16:35:19.298910 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 16:35:19.301769 5991 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:35:19.304796 5991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:35:19.319708 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.313839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.27.122.194:43619)
0928 16:35:19.314262 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:19.314274 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:19.314891 (+ 617us) server_negotiation.cc:407] Connection header received
0928 16:35:19.314968 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:19.314973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:19.315136 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:19.315276 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:35:19.315702 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.316689 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.318044 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.318887 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.319012 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.319157 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:19.319442 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 16:35:19.319552 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":107,"threads_started":1}
I20260928 16:35:19.319733 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.313087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.27.122.193:44621)
0928 16:35:19.313398 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:19.313411 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:19.314175 (+ 764us) server_negotiation.cc:407] Connection header received
0928 16:35:19.314527 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:19.314535 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:19.314754 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:19.314908 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:35:19.315337 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.316190 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.318044 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.318887 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.319012 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.319157 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:19.319497 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 16:35:19.319587 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":126,"threads_started":1}
I20260928 16:35:19.319746 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.313643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:43619 (local address 127.0.0.1:59310)
0928 16:35:19.314695 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:19.314708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:19.314791 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:19.315443 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:19.315445 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:19.315462 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:35:19.315560 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.315566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.316786 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.316789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.317937 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.317946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.319023 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.319027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.319159 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.319209 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 16:35:19.319368 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 16:35:19.319594 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"thread_start_us":96,"threads_started":1}
I20260928 16:35:19.320099 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.312963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:44621 (local address 127.0.0.1:42336)
0928 16:35:19.314020 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:19.314069 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:19.314375 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:19.315069 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:19.315072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:19.315089 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:35:19.315204 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.315214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.316323 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.316326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.317883 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.317894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.319226 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.319231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.319336 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.319360 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:35:19.319881 (+ 521us) client_negotiation.cc:240] Negotiation successful
0928 16:35:19.319980 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":48,"thread_start_us":60,"threads_started":1}
I20260928 16:35:19.320585 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.313152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:46843 (local address 127.0.0.1:41586)
0928 16:35:19.314097 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:19.314113 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:19.314243 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:19.314960 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:19.314970 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:19.314992 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:35:19.315121 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.315128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.316402 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.316405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.318650 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.318657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.320034 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.320037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.320169 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.320199 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:35:19.320387 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 16:35:19.320461 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":1379,"thread_start_us":111,"threads_started":1}
I20260928 16:35:19.321197 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.313488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41586 (local address 127.27.122.195:46843)
0928 16:35:19.313874 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:19.313885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:19.314389 (+ 504us) server_negotiation.cc:407] Connection header received
0928 16:35:19.314472 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:19.314477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:19.314788 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:19.314964 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:35:19.315251 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.316270 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.318766 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.319709 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.319832 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.319969 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:19.320917 (+ 948us) server_negotiation.cc:299] Negotiation successful
0928 16:35:19.321047 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":161,"threads_started":1}
I20260928 16:35:19.323009 6253 tablet_service.cc:1467] Processing CreateTablet for tablet dfb77fe149c84e8b8e2934c1d331fb43 (DEFAULT_TABLE table=test-table [id=cc4e88e0e3ac4a49bd392691ac2a7a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:35:19.323386 6389 tablet_service.cc:1467] Processing CreateTablet for tablet dfb77fe149c84e8b8e2934c1d331fb43 (DEFAULT_TABLE table=test-table [id=cc4e88e0e3ac4a49bd392691ac2a7a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:35:19.323997 6253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb77fe149c84e8b8e2934c1d331fb43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:35:19.324350 6389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb77fe149c84e8b8e2934c1d331fb43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:35:19.331041 6117 tablet_service.cc:1467] Processing CreateTablet for tablet dfb77fe149c84e8b8e2934c1d331fb43 (DEFAULT_TABLE table=test-table [id=cc4e88e0e3ac4a49bd392691ac2a7a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:35:19.331984 6117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfb77fe149c84e8b8e2934c1d331fb43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:35:19.332649 6472 tablet_bootstrap.cc:492] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb: Bootstrap starting.
I20260928 16:35:19.334998 6472 tablet_bootstrap.cc:654] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb: Neither blocks nor log segments found. Creating new log.
I20260928 16:35:19.336331 6472 log.cc:824] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb: Log is configured to *not* fsync() on all Append() calls
I20260928 16:35:19.339110 6474 tablet_bootstrap.cc:492] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823: Bootstrap starting.
I20260928 16:35:19.340051 6475 tablet_bootstrap.cc:492] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c: Bootstrap starting.
I20260928 16:35:19.341378 6474 tablet_bootstrap.cc:654] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823: Neither blocks nor log segments found. Creating new log.
I20260928 16:35:19.341676 6472 tablet_bootstrap.cc:492] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb: No bootstrap required, opened a new log
I20260928 16:35:19.341877 6472 ts_tablet_manager.cc:1397] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 16:35:19.342654 6475 tablet_bootstrap.cc:654] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c: Neither blocks nor log segments found. Creating new log.
I20260928 16:35:19.342685 6474 log.cc:824] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823: Log is configured to *not* fsync() on all Append() calls
I20260928 16:35:19.343497 6475 log.cc:824] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:35:19.345162 6474 tablet_bootstrap.cc:492] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823: No bootstrap required, opened a new log
I20260928 16:35:19.345194 6475 tablet_bootstrap.cc:492] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c: No bootstrap required, opened a new log
I20260928 16:35:19.345346 6474 ts_tablet_manager.cc:1397] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:35:19.345379 6475 ts_tablet_manager.cc:1397] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:35:19.345595 6472 raft_consensus.cc:348] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d381b12661a48ba8aede1b2416ed823" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 44621 } } peers { permanent_uuid: "a287305093704ecba375ea243fb77bfb" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46843 } } peers { permanent_uuid: "4fc7a86e2271467fa0a76af69137296c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43619 } }
I20260928 16:35:19.345857 6472 raft_consensus.cc:374] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:35:19.345922 6472 raft_consensus.cc:729] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a287305093704ecba375ea243fb77bfb, State: Initialized, Role: FOLLOWER
I20260928 16:35:19.346392 6472 consensus_queue.cc:260] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d381b12661a48ba8aede1b2416ed823" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 44621 } } peers { permanent_uuid: "a287305093704ecba375ea243fb77bfb" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46843 } } peers { permanent_uuid: "4fc7a86e2271467fa0a76af69137296c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43619 } }
I20260928 16:35:19.347044 6455 heartbeater.cc:503] Master 127.27.122.254:34081 was elected leader, sending a full tablet report...
I20260928 16:35:19.347381 6472 ts_tablet_manager.cc:1428] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 16:35:19.348891 6474 raft_consensus.cc:348] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d381b12661a48ba8aede1b2416ed823" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 44621 } } peers { permanent_uuid: "a287305093704ecba375ea243fb77bfb" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46843 } } peers { permanent_uuid: "4fc7a86e2271467fa0a76af69137296c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43619 } }
I20260928 16:35:19.348961 6475 raft_consensus.cc:348] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d381b12661a48ba8aede1b2416ed823" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 44621 } } peers { permanent_uuid: "a287305093704ecba375ea243fb77bfb" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46843 } } peers { permanent_uuid: "4fc7a86e2271467fa0a76af69137296c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43619 } }
I20260928 16:35:19.349176 6474 raft_consensus.cc:374] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:35:19.349215 6475 raft_consensus.cc:374] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:35:19.349226 6474 raft_consensus.cc:729] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d381b12661a48ba8aede1b2416ed823, State: Initialized, Role: FOLLOWER
I20260928 16:35:19.349287 6475 raft_consensus.cc:729] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc7a86e2271467fa0a76af69137296c, State: Initialized, Role: FOLLOWER
I20260928 16:35:19.349637 6474 consensus_queue.cc:260] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d381b12661a48ba8aede1b2416ed823" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 44621 } } peers { permanent_uuid: "a287305093704ecba375ea243fb77bfb" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46843 } } peers { permanent_uuid: "4fc7a86e2271467fa0a76af69137296c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43619 } }
I20260928 16:35:19.349784 6475 consensus_queue.cc:260] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d381b12661a48ba8aede1b2416ed823" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 44621 } } peers { permanent_uuid: "a287305093704ecba375ea243fb77bfb" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46843 } } peers { permanent_uuid: "4fc7a86e2271467fa0a76af69137296c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43619 } }
I20260928 16:35:19.350591 6474 ts_tablet_manager.cc:1428] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:35:19.350703 6475 ts_tablet_manager.cc:1428] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:35:19.350924 6184 heartbeater.cc:503] Master 127.27.122.254:34081 was elected leader, sending a full tablet report...
I20260928 16:35:19.351025 6320 heartbeater.cc:503] Master 127.27.122.254:34081 was elected leader, sending a full tablet report...
I20260928 16:35:19.402657 6478 raft_consensus.cc:482] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:35:19.402889 6478 raft_consensus.cc:504] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d381b12661a48ba8aede1b2416ed823" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 44621 } } peers { permanent_uuid: "a287305093704ecba375ea243fb77bfb" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46843 } } peers { permanent_uuid: "4fc7a86e2271467fa0a76af69137296c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43619 } }
I20260928 16:35:19.404049 6478 leader_election.cc:288] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d381b12661a48ba8aede1b2416ed823 (127.27.122.193:44621), 4fc7a86e2271467fa0a76af69137296c (127.27.122.194:43619)
I20260928 16:35:19.408941 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.404387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:51519 (local address 127.27.122.193:44621)
0928 16:35:19.404536 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:19.404543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:19.404562 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:35:19.404646 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:19.404649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:19.404708 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:19.404776 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:35:19.405153 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.405936 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.407592 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.408189 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.408314 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.408367 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:19.408714 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 16:35:19.408794 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:35:19.408954 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.404379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:49099 (local address 127.27.122.194:43619)
0928 16:35:19.404509 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:19.404513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:19.404884 (+ 371us) server_negotiation.cc:407] Connection header received
0928 16:35:19.404998 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:19.405001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:19.405045 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:19.405140 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:35:19.405384 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.406082 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.407592 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.408202 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.408320 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.408367 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:19.408752 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 16:35:19.408824 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:35:19.408950 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.404262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:44621 (local address 127.27.122.195:51519)
0928 16:35:19.404411 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:19.404427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:19.404526 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:19.404818 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:19.404823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:19.404849 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:35:19.405044 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.405050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.406053 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.406055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.407464 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.407470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.408320 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.408324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.408493 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.408600 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 16:35:19.408672 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 16:35:19.408758 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":152,"spinlock_wait_cycles":8448}
I20260928 16:35:19.408980 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.404262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:43619 (local address 127.27.122.195:49099)
0928 16:35:19.404774 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:19.404790 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:35:19.404916 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:35:19.405146 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:35:19.405150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:35:19.405172 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:35:19.405261 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.405265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.406178 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.406181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.407458 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:35:19.407470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.408320 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.408324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:35:19.408518 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.408596 (+ 78us) client_negotiation.cc:769] Sending connection context
0928 16:35:19.408726 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 16:35:19.408773 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":207,"spinlock_wait_cycles":1152,"thread_start_us":149,"threads_started":1}
I20260928 16:35:19.409698 6274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb77fe149c84e8b8e2934c1d331fb43" candidate_uuid: "a287305093704ecba375ea243fb77bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc7a86e2271467fa0a76af69137296c" is_pre_election: true
I20260928 16:35:19.409806 6137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb77fe149c84e8b8e2934c1d331fb43" candidate_uuid: "a287305093704ecba375ea243fb77bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d381b12661a48ba8aede1b2416ed823" is_pre_election: true
I20260928 16:35:19.410068 6274 raft_consensus.cc:2454] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a287305093704ecba375ea243fb77bfb in term 0.
I20260928 16:35:19.410095 6137 raft_consensus.cc:2454] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a287305093704ecba375ea243fb77bfb in term 0.
I20260928 16:35:19.410561 6345 leader_election.cc:302] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc7a86e2271467fa0a76af69137296c, a287305093704ecba375ea243fb77bfb; no voters:
I20260928 16:35:19.410836 6478 raft_consensus.cc:2781] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:35:19.410928 6478 raft_consensus.cc:482] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:35:19.410980 6478 raft_consensus.cc:3037] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:35:19.411978 6478 raft_consensus.cc:504] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d381b12661a48ba8aede1b2416ed823" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 44621 } } peers { permanent_uuid: "a287305093704ecba375ea243fb77bfb" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46843 } } peers { permanent_uuid: "4fc7a86e2271467fa0a76af69137296c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43619 } }
I20260928 16:35:19.412290 6478 leader_election.cc:288] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [CANDIDATE]: Term 1 election: Requested vote from peers 1d381b12661a48ba8aede1b2416ed823 (127.27.122.193:44621), 4fc7a86e2271467fa0a76af69137296c (127.27.122.194:43619)
I20260928 16:35:19.412611 6274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb77fe149c84e8b8e2934c1d331fb43" candidate_uuid: "a287305093704ecba375ea243fb77bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc7a86e2271467fa0a76af69137296c"
I20260928 16:35:19.412617 6137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfb77fe149c84e8b8e2934c1d331fb43" candidate_uuid: "a287305093704ecba375ea243fb77bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d381b12661a48ba8aede1b2416ed823"
I20260928 16:35:19.412730 6137 raft_consensus.cc:3037] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:35:19.412729 6274 raft_consensus.cc:3037] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:35:19.414009 6274 raft_consensus.cc:2454] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a287305093704ecba375ea243fb77bfb in term 1.
I20260928 16:35:19.414009 6137 raft_consensus.cc:2454] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a287305093704ecba375ea243fb77bfb in term 1.
I20260928 16:35:19.414270 6345 leader_election.cc:302] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc7a86e2271467fa0a76af69137296c, a287305093704ecba375ea243fb77bfb; no voters:
I20260928 16:35:19.414443 6478 raft_consensus.cc:2781] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:35:19.414706 6478 raft_consensus.cc:686] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [term 1 LEADER]: Becoming Leader. State: Replica: a287305093704ecba375ea243fb77bfb, State: Running, Role: LEADER
I20260928 16:35:19.415019 6478 consensus_queue.cc:237] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d381b12661a48ba8aede1b2416ed823" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 44621 } } peers { permanent_uuid: "a287305093704ecba375ea243fb77bfb" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46843 } } peers { permanent_uuid: "4fc7a86e2271467fa0a76af69137296c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43619 } }
I20260928 16:35:19.418399 5991 catalog_manager.cc:5495] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to a287305093704ecba375ea243fb77bfb (127.27.122.195). New cstate: current_term: 1 leader_uuid: "a287305093704ecba375ea243fb77bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d381b12661a48ba8aede1b2416ed823" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 44621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a287305093704ecba375ea243fb77bfb" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 46843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc7a86e2271467fa0a76af69137296c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43619 } health_report { overall_health: UNKNOWN } } }
I20260928 16:35:19.447880 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.444598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.27.122.195:46843)
0928 16:35:19.444733 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:35:19.444737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:35:19.444751 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:35:19.444876 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:35:19.444879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:35:19.444959 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:35:19.445054 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:35:19.445257 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.445780 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.446623 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:35:19.446828 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:35:19.446904 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:35:19.447157 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:35:19.447574 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:35:19.447680 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:35:19.447724 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 16:35:19.447772 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:35:19.453712 6274 raft_consensus.cc:1261] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c [term 1 FOLLOWER]: Refusing update from remote peer a287305093704ecba375ea243fb77bfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:35:19.453713 6137 raft_consensus.cc:1261] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823 [term 1 FOLLOWER]: Refusing update from remote peer a287305093704ecba375ea243fb77bfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:35:19.454425 6478 consensus_queue.cc:1035] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d381b12661a48ba8aede1b2416ed823" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 44621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:35:19.454609 6482 consensus_queue.cc:1035] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc7a86e2271467fa0a76af69137296c" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 43619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:35:19.462185 6488 mvcc.cc:205] Tried to move back new op lower bound from 7334352156474798080 to 7334352156326924288. Current Snapshot: MvccSnapshot[applied={T|T < 7334352156474798080}]
I20260928 16:35:19.470077 5990 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:35:19.470212 5990 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:35:19.472363 5990 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2daf5c4f71ff425388cef7ad3867d268: Sending DeleteTablet for 3 replicas of tablet dfb77fe149c84e8b8e2934c1d331fb43
I20260928 16:35:19.473107 6117 tablet_service.cc:1514] Processing DeleteTablet for tablet dfb77fe149c84e8b8e2934c1d331fb43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:35:19 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:42336
I20260928 16:35:19.473201 6253 tablet_service.cc:1514] Processing DeleteTablet for tablet dfb77fe149c84e8b8e2934c1d331fb43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:35:19 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:59310
I20260928 16:35:19.473201 6389 tablet_service.cc:1514] Processing DeleteTablet for tablet dfb77fe149c84e8b8e2934c1d331fb43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:35:19 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:41586
I20260928 16:35:19.473569 6495 tablet_replica.cc:330] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823: stopping tablet replica
I20260928 16:35:19.473701 6497 tablet_replica.cc:330] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb: stopping tablet replica
I20260928 16:35:19.473889 6495 raft_consensus.cc:2229] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:35:19.474014 6497 raft_consensus.cc:2229] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:35:19.474181 6495 raft_consensus.cc:2258] T dfb77fe149c84e8b8e2934c1d331fb43 P 1d381b12661a48ba8aede1b2416ed823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:35:19.474362 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 6054
I20260928 16:35:19.475044 6497 raft_consensus.cc:2258] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:35:19.475442 6496 tablet_replica.cc:330] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c: stopping tablet replica
I20260928 16:35:19.475698 6496 raft_consensus.cc:2229] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:35:19.475948 6496 raft_consensus.cc:2258] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:35:19.477380 6496 ts_tablet_manager.cc:1905] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:35:19.477380 6497 ts_tablet_manager.cc:1905] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:35:19.480074 6496 ts_tablet_manager.cc:1918] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:35:19.480074 6497 ts_tablet_manager.cc:1918] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:35:19.480175 6497 log.cc:1196] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/dfb77fe149c84e8b8e2934c1d331fb43
I20260928 16:35:19.480288 6496 log.cc:1196] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/dfb77fe149c84e8b8e2934c1d331fb43
I20260928 16:35:19.480522 6497 ts_tablet_manager.cc:1939] T dfb77fe149c84e8b8e2934c1d331fb43 P a287305093704ecba375ea243fb77bfb: Deleting consensus metadata
I20260928 16:35:19.480749 6496 ts_tablet_manager.cc:1939] T dfb77fe149c84e8b8e2934c1d331fb43 P 4fc7a86e2271467fa0a76af69137296c: Deleting consensus metadata
I20260928 16:35:19.483042 5978 catalog_manager.cc:4841] TS 4fc7a86e2271467fa0a76af69137296c (127.27.122.194:43619): tablet dfb77fe149c84e8b8e2934c1d331fb43 (table test-table [id=cc4e88e0e3ac4a49bd392691ac2a7a4c]) successfully deleted
I20260928 16:35:19.483238 5978 catalog_manager.cc:4841] TS a287305093704ecba375ea243fb77bfb (127.27.122.195:46843): tablet dfb77fe149c84e8b8e2934c1d331fb43 (table test-table [id=cc4e88e0e3ac4a49bd392691ac2a7a4c]) successfully deleted
W20260928 16:35:19.486912 5975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:44621 (error 108)
I20260928 16:35:19.487066 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 6189
I20260928 16:35:19.489482 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:35:19.489138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:44621 (local address 127.0.0.1:58374)
0928 16:35:19.489296 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 16:35:19.489376 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:44621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 16:35:19.489717 5975 catalog_manager.cc:4584] TS 1d381b12661a48ba8aede1b2416ed823 (127.27.122.193:44621): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfb77fe149c84e8b8e2934c1d331fb43: Network error: Client connection negotiation failed: client connection to 127.27.122.193:44621: connect: Connection refused (error 111)
I20260928 16:35:19.493693 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 6325
I20260928 16:35:19.498999 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 5959
2026-09-28T16:35:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2275 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:35:19 dist-test-slave-g9sh krb5kdc[6501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:35:19 dist-test-slave-g9sh krb5kdc[6501](info): set up 2 sockets
Sep 28 16:35:19 dist-test-slave-g9sh krb5kdc[6501](info): commencing operation
krb5kdc: starting...
W20260928 16:35:20.532646 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:35:20 dist-test-slave-g9sh krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:35:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:35:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.122.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.122.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/rangeradmin_127.27.122.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/rangeradmin_127.27.122.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.122.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.122.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/rangerlookup_127.27.122.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/rangerlookup_127.27.122.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.122.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.122.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/HTTP_127.27.122.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/HTTP_127.27.122.212@KRBTEST.COM.keytab.
I20260928 16:35:20.711531 28139 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYGQpme/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/postgres -l logfile start
2026-09-28 16:35:23.171 UTC [6531] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 16:35:23.171 UTC [6531] LOG: listening on IPv4 address "127.27.122.212", port 47967
2026-09-28 16:35:23.175 UTC [6531] LOG: listening on Unix socket "/tmp/.s.PGSQL.47967"
2026-09-28 16:35:23.209 UTC [6534] LOG: database system was shut down at 2026-09-28 16:35:21 UTC
2026-09-28 16:35:23.213 UTC [6531] LOG: database system is ready to accept connections
I20260928 16:35:24.155968 28139 mini_postgres.cc:96] Postgres bound to 47967
2026-09-28 16:35:24.163 UTC [6542] FATAL: database "slave" does not exist
127.27.122.212:47967 - accepting connections
I20260928 16:35:24.164459 28139 mini_ranger.cc:161] Starting Ranger...
I20260928 16:35:24.184648 28139 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 16:35:24.617187 28139 mini_ranger.cc:87] Created ranger Postgres database
I20260928 16:35:24.617290 28139 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-admin
2026-09-28 16:35:25,097 [I] DB FLAVOR :POSTGRES
2026-09-28 16:35:25,097 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:35:25,097 [I] Checking connection..
2026-09-28 16:35:25,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 16:35:25,375 [I] Checking connection passed.
2026-09-28 16:35:25,375 [I] --------- Verifying version history table ---------
2026-09-28 16:35:25,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:35:25,623 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:35:25,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:35:25,884 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:35:25,884 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 16:35:25,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 16:35:26.135 UTC [6606] WARNING: there is no transaction in progress
2026-09-28 16:35:26,141 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 16:35:26,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:35:26,408 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 16:35:26,409 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:35:26,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 16:35:26,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 16:35:26,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g9sh.c.gcp-upstream.internal','N') ;"
2026-09-28 16:35:27,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:35:27,395 [I] Table x_portal_user does not exist in database ranger
2026-09-28 16:35:27,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:35:27,652 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 16:35:27,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:35:27,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:35:28,158 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 16:35:28,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 16:35:29.091 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.107 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.124 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.139 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.157 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.177 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.230 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.237 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.251 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.261 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.271 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.284 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.293 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.299 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.305 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29.311 UTC [6715] WARNING: there is no transaction in progress
2026-09-28 16:35:29,929 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 16:35:29,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 16:35:30,167 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 16:35:30,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:35:30,434 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 16:35:30,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:35:30,685 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 16:35:30,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:35:30,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:35:31,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g9sh.c.gcp-upstream.internal';"
2026-09-28 16:35:31,407 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 16:35:31,407 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 16:35:31,655 [I] DB_PATCHES have already been applied
I20260928 16:35:31.661170 28139 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYGQpme/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 16:35:31.661238 28139 mini_ranger.cc:193] Using host: 127.27.122.212
I20260928 16:35:31.664695 28139 mini_ranger.cc:227] Ranger admin URL: http://127.27.122.212:34377
Sep 28, 2026 4:35:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:35:32 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:35:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:35:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:35:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:35:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:35:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 16:35:33.929726 28139 mini_ranger.cc:160] Time spent starting Ranger: real 9.765s user 0.004s sys 0.008s
Sep 28, 2026 4:35:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:35:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 4:35:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:35:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:35:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:35:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:35:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:35:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:35:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:35:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:35:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:35:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:35:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:35:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:35:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:35:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:35:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:35:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 16:35:56.437613 28139 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.122.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.122.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/rangerkms_127.27.122.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/rangerkms_127.27.122.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.122.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.122.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.122.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/HTTP_127.27.122.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.122.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/HTTP_127.27.122.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 16:35:56 dist-test-slave-g9sh krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613356, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 16:35:56.531976 28139 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 16:35:56.541536 28139 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 16:35:57.337687 28139 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 16:35:57.337791 28139 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-kms
I20260928 16:35:57.337909 28139 mini_ranger_kms.cc:227] Using postgres at 127.27.122.212:47967
2026-09-28 16:35:57,956 [I] DB FLAVOR :POSTGRES
2026-09-28 16:35:57,956 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:35:57,956 [I] Checking connection
2026-09-28 16:35:57,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 16:35:58,206 [I] connection success
2026-09-28 16:35:58,207 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 16:35:58,207 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 16:35:58,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 16:35:58,468 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 16:35:58,468 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:35:58,468 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 16:35:58,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:47967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 16:35:58,754 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 16:35:59.287919 28139 mini_ranger_kms.cc:310] Created kms service
I20260928 16:35:59.510954 28139 mini_ranger_kms.cc:326] Created kudu user
I20260928 16:35:59.555325 28139 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 16:35:59 dist-test-slave-g9sh krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613356, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:35:59.965067 28139 mini_ranger_kms.cc:384] Added ranger policy
I20260928 16:35:59.965229 28139 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 16:35:59.965256 28139 mini_ranger_kms.cc:240] Using host: 127.27.122.212
I20260928 16:35:59.968978 28139 mini_ranger_kms.cc:276] Ranger KMS PID: 6898
I20260928 16:35:59.969122 28139 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.122.212:42373
Sep 28, 2026 4:36:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:36:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:36:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:36:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:36:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:36:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 16:36:02.201689 28139 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.670s user 0.030s sys 0.260s
I20260928 16:36:02.201825 28139 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 16:36:02.201953 28139 mini_ranger_kms.cc:401] 127.27.122.212:42373/kms/v1/keys
Sep 28 16:36:02 dist-test-slave-g9sh krb5kdc[6501](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613362, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.122.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 16:36:02 dist-test-slave-g9sh krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613362, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.122.212@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
Sep 28 16:36:03 dist-test-slave-g9sh krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613362, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.122.212@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
Sep 28 16:36:05 dist-test-slave-g9sh krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:36:05.528432 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:35685
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:35685
--ranger_config_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.122.212:42373/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:36:05.665290 6949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:05.665575 6949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:05.665661 6949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:05.669689 6949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:36:05.669770 6949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:05.669806 6949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:05.669860 6949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:36:05.669899 6949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:36:05.675365 6949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.122.212:42373/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:35685
--ranger_config_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:35685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:05.676419 6949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:05.677561 6949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:05.687801 6954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:05.687867 6949 server_base.cc:1034] running on GCE node
W20260928 16:36:05.688084 6957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:05.687819 6955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:05.688653 6949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:05.689741 6949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:05.690927 6949 hybrid_clock.cc:648] HybridClock initialized: now 1790613365690910 us; error 46 us; skew 500 ppm
Sep 28 16:36:05 dist-test-slave-g9sh krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613365, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:05.693782 6949 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:36:05.694669 6949 webserver.cc:460] Webserver started at http://127.27.122.254:39641/ using document root <none> and password file <none>
I20260928 16:36:05.695187 6949 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:05.695241 6949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:36:05.695472 6949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:36:05 dist-test-slave-g9sh krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613365, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:05.751209 6949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "1810dccc3afd4a02add0ce6a8bd3035b"
format_stamp: "Formatted at 2026-09-28 16:36:05 on dist-test-slave-g9sh"
server_key: "9c968cb99f2d3bbbcae947a58b548da2"
server_key_iv: "d06292e52e7a0e8492a0ef45d111667b"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:05.751982 6949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "1810dccc3afd4a02add0ce6a8bd3035b"
format_stamp: "Formatted at 2026-09-28 16:36:05 on dist-test-slave-g9sh"
server_key: "9c968cb99f2d3bbbcae947a58b548da2"
server_key_iv: "d06292e52e7a0e8492a0ef45d111667b"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:05.756834 6949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 16:36:05.773195 6964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:05.774819 6949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:36:05.774964 6949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "1810dccc3afd4a02add0ce6a8bd3035b"
format_stamp: "Formatted at 2026-09-28 16:36:05 on dist-test-slave-g9sh"
server_key: "9c968cb99f2d3bbbcae947a58b548da2"
server_key_iv: "d06292e52e7a0e8492a0ef45d111667b"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:05.775097 6949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:05.792111 6949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:05.795368 6949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:05.795589 6949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:05.805420 6949 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:35685
I20260928 16:36:05.805415 7017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:35685 every 8 connection(s)
I20260928 16:36:05.806689 6949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:36:05.809651 7018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:36:05.815061 7018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b: Bootstrap starting.
I20260928 16:36:05.815864 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 6949
I20260928 16:36:05.815996 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:36:05.816305 28139 external_mini_cluster.cc:1428] Setting key b6bca693b5071191e0c36d8fa17ea788
I20260928 16:36:05.818392 7018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b: Neither blocks nor log segments found. Creating new log.
I20260928 16:36:05.819144 7018 log.cc:824] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:36:05.821341 7018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b: No bootstrap required, opened a new log
I20260928 16:36:05.825305 7018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1810dccc3afd4a02add0ce6a8bd3035b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35685 } }
I20260928 16:36:05.825626 7018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:36:05.825739 7018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1810dccc3afd4a02add0ce6a8bd3035b, State: Initialized, Role: FOLLOWER
I20260928 16:36:05.826280 7018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1810dccc3afd4a02add0ce6a8bd3035b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35685 } }
I20260928 16:36:05.826452 7018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:36:05.826574 7018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:36:05.826691 7018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [term 0 FOLLOWER]: Advancing to term 1
Sep 28 16:36:05 dist-test-slave-g9sh krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:36:05.827816 7018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1810dccc3afd4a02add0ce6a8bd3035b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35685 } }
I20260928 16:36:05.828169 7018 leader_election.cc:302] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1810dccc3afd4a02add0ce6a8bd3035b; no voters:
I20260928 16:36:05.828536 7018 leader_election.cc:288] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:36:05.828612 7023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:36:05.828846 7023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [term 1 LEADER]: Becoming Leader. State: Replica: 1810dccc3afd4a02add0ce6a8bd3035b, State: Running, Role: LEADER
I20260928 16:36:05.829448 7023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1810dccc3afd4a02add0ce6a8bd3035b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35685 } }
I20260928 16:36:05.830745 7025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1810dccc3afd4a02add0ce6a8bd3035b. Latest consensus state: current_term: 1 leader_uuid: "1810dccc3afd4a02add0ce6a8bd3035b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1810dccc3afd4a02add0ce6a8bd3035b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35685 } } }
I20260928 16:36:05.830952 7025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [sys.catalog]: This master's current role is: LEADER
I20260928 16:36:05.831348 7018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:36:05.831563 7027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:36:05.833484 7024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1810dccc3afd4a02add0ce6a8bd3035b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1810dccc3afd4a02add0ce6a8bd3035b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35685 } } }
I20260928 16:36:05.833667 7024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b [sys.catalog]: This master's current role is: LEADER
I20260928 16:36:05.833827 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:05.817861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58848 (local address 127.27.122.254:35685)
0928 16:36:05.818351 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:05.818378 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:05.818422 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:36:05.819288 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:05.819312 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:05.819752 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:05.820115 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:05.821004 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:05.822622 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:05.823888 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:05.824613 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:05.824780 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:05.827826 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:05.827833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:05.827849 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:05.827880 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:05.830037 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:05.830565 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:05.830571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:05.830576 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:05.830713 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:05.831041 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:05.831046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:05.831048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:05.831619 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:05.831804 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:05.832630 (+ 826us) server_negotiation.cc:299] Negotiation successful
0928 16:36:05.832947 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":151,"threads_started":1}
I20260928 16:36:05.834766 7027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:36:05.835264 7018 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:36:05.844269 7040 mvcc.cc:205] Tried to move back new op lower bound from 7334352346488885248 to 7334352346438721536. Current Snapshot: MvccSnapshot[applied={T|T < 7334352346488885248}]
I20260928 16:36:05.846100 7027 catalog_manager.cc:1347] Generated new cluster ID: 209dbd803123490bbc0236a191b6f41a
I20260928 16:36:05.846163 7027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:36:05.860215 7027 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:36:05.861192 7027 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:36:05.872457 7027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b: Generated new TSK 0
I20260928 16:36:05.873557 7027 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:36:06.584977 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.122.212:42373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35685
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:36:06.710357 7063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:06.710691 7063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:06.710790 7063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:06.715472 7063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:06.715562 7063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:06.715690 7063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:36:06.721292 7063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.122.212:42373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:06.722436 7063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:06.723481 7063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:06.734028 7069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:06.734396 7071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:06.734592 7068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:06.735222 7063 server_base.cc:1034] running on GCE node
I20260928 16:36:06.735675 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:06.736363 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:06.737556 7063 hybrid_clock.cc:648] HybridClock initialized: now 1790613366737534 us; error 34 us; skew 500 ppm
Sep 28 16:36:06 dist-test-slave-g9sh krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613366, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:06.741223 7063 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:36:06.742527 7063 webserver.cc:460] Webserver started at http://127.27.122.193:34675/ using document root <none> and password file <none>
I20260928 16:36:06.743173 7063 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:06.743271 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:36:06.743608 7063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:36:06 dist-test-slave-g9sh krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613366, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:06.759487 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "af719e503f6545ebb5217ad1fc001aff"
format_stamp: "Formatted at 2026-09-28 16:36:06 on dist-test-slave-g9sh"
server_key: "69e0d65fc8449ab7f90403f6b0f3189d"
server_key_iv: "339511fcac83a69a2a9c8259a0eb944f"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:06.760324 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "af719e503f6545ebb5217ad1fc001aff"
format_stamp: "Formatted at 2026-09-28 16:36:06 on dist-test-slave-g9sh"
server_key: "69e0d65fc8449ab7f90403f6b0f3189d"
server_key_iv: "339511fcac83a69a2a9c8259a0eb944f"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:06.766386 7063 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 16:36:06.789377 7080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:06.791147 7063 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260928 16:36:06.791280 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "af719e503f6545ebb5217ad1fc001aff"
format_stamp: "Formatted at 2026-09-28 16:36:06 on dist-test-slave-g9sh"
server_key: "69e0d65fc8449ab7f90403f6b0f3189d"
server_key_iv: "339511fcac83a69a2a9c8259a0eb944f"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:06.791419 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:06.809811 7063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:06.813287 7063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:06.813506 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:06.814127 7063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:36:06.815147 7063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:36:06.815253 7063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:06.815372 7063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:36:06.815442 7063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:06.829773 7063 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:35309
I20260928 16:36:06.831040 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:36:06.832460 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 7063
I20260928 16:36:06.832592 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:36:06.832935 28139 external_mini_cluster.cc:1428] Setting key 43cafc75e26eb09dd32e29dc9ad932b7
I20260928 16:36:06.841259 7194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:35309 every 8 connection(s)
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
Sep 28 16:36:06 dist-test-slave-g9sh krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613366, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
I20260928 16:36:06.877434 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:06.852787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:36055 (local address 127.27.122.254:35685)
0928 16:36:06.859927 (+ 7140us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:06.859936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:06.861923 (+ 1987us) server_negotiation.cc:407] Connection header received
0928 16:36:06.861980 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:06.861983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:06.862066 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:06.862173 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:06.863731 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:06.864661 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:06.866522 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:06.866935 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:06.867034 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:06.872069 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:06.872073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:06.872076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:06.872105 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:06.874487 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:06.875166 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:06.875170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:06.875172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:06.875267 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:06.875673 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:06.875676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:06.875678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:06.875908 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:06.876014 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:06.877094 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0928 16:36:06.877228 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7045,"thread_start_us":130,"threads_started":1}
I20260928 16:36:06.878060 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:06.851539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35685 (local address 127.27.122.193:36055)
0928 16:36:06.860446 (+ 8907us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:06.860511 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:06.861601 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:06.862514 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:06.862524 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:06.862875 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:06.863556 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:06.863576 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:06.864831 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:06.864834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:06.866364 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:06.866376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:06.867257 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:06.867261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:06.867538 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:06.868421 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:06.868443 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:06.871841 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:06.874643 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:06.874652 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:06.874674 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:06.875043 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:06.875369 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:06.875372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:06.875374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:06.875558 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:06.876072 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:06.876079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:06.876396 (+ 317us) client_negotiation.cc:769] Sending connection context
0928 16:36:06.876590 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 16:36:06.876950 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8634,"thread_start_us":171,"threads_started":1}
I20260928 16:36:06.879966 7195 heartbeater.cc:347] Connected to a master server at 127.27.122.254:35685
I20260928 16:36:06.880214 7195 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:06.880822 7195 heartbeater.cc:511] Master 127.27.122.254:35685 requested a full tablet report, sending...
I20260928 16:36:06.882851 6981 ts_manager.cc:194] Registered new tserver with Master: af719e503f6545ebb5217ad1fc001aff (127.27.122.193:35309)
I20260928 16:36:06.884438 6981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:36055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:36:06.900563 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.122.212:42373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35685
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:36:06 dist-test-slave-g9sh krb5kdc[6501](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613366, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 16:36:07.067411 7203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:07.067667 7203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:07.067734 7203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:07.071733 7203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:07.071818 7203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:07.071919 7203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:36:07.077219 7203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.122.212:42373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:07.078426 7203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:07.079820 7203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:07.091656 7212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:07.091785 7210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:07.091622 7209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:07.092199 7203 server_base.cc:1034] running on GCE node
I20260928 16:36:07.092748 7203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:07.093556 7203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:07.094758 7203 hybrid_clock.cc:648] HybridClock initialized: now 1790613367094734 us; error 37 us; skew 500 ppm
Sep 28 16:36:07 dist-test-slave-g9sh krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613367, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:07.098827 7203 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:36:07.099882 7203 webserver.cc:460] Webserver started at http://127.27.122.194:33697/ using document root <none> and password file <none>
I20260928 16:36:07.100590 7203 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:07.100683 7203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:36:07.100986 7203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:36:07 dist-test-slave-g9sh krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613367, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:07.126636 7203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "4d89faa9a77d4a6098fb35c020b19ea6"
format_stamp: "Formatted at 2026-09-28 16:36:07 on dist-test-slave-g9sh"
server_key: "f470df78b9bef8c7599ccf63ce75427b"
server_key_iv: "5c5cad2975ebc04363e9c9528b7d9ba4"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:07.127455 7203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "4d89faa9a77d4a6098fb35c020b19ea6"
format_stamp: "Formatted at 2026-09-28 16:36:07 on dist-test-slave-g9sh"
server_key: "f470df78b9bef8c7599ccf63ce75427b"
server_key_iv: "5c5cad2975ebc04363e9c9528b7d9ba4"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:07.133342 7203 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 16:36:07.155934 7219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:07.157713 7203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:36:07.157845 7203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "4d89faa9a77d4a6098fb35c020b19ea6"
format_stamp: "Formatted at 2026-09-28 16:36:07 on dist-test-slave-g9sh"
server_key: "f470df78b9bef8c7599ccf63ce75427b"
server_key_iv: "5c5cad2975ebc04363e9c9528b7d9ba4"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:07.157986 7203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:07.183485 7203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:07.187386 7203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:07.187677 7203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:07.188443 7203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:36:07.189651 7203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:36:07.189764 7203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:07.189868 7203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:36:07.189934 7203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:07.203408 7203 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:45217
I20260928 16:36:07.204809 7203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:36:07.209903 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 7203
I20260928 16:36:07.210058 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:36:07.210501 28139 external_mini_cluster.cc:1428] Setting key de5af5529394d2ed73b6e549e45f6851
I20260928 16:36:07.217217 7333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:45217 every 8 connection(s)
Sep 28 16:36:07 dist-test-slave-g9sh krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613367, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
I20260928 16:36:07.236809 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.221168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:54755 (local address 127.27.122.254:35685)
0928 16:36:07.221306 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:07.221311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:07.221797 (+ 486us) server_negotiation.cc:407] Connection header received
0928 16:36:07.223155 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:07.223160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:07.223220 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:07.223278 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:07.225251 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.226210 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.228048 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.228399 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.228490 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.231981 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:07.231985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:07.231988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:07.232017 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:07.234004 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:07.234685 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:07.234689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:07.234691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:07.234788 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:07.235216 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:07.235219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:07.235221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:07.235457 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:07.235555 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:07.236406 (+ 851us) server_negotiation.cc:299] Negotiation successful
0928 16:36:07.236604 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
I20260928 16:36:07.238869 7337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.219948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35685 (local address 127.27.122.194:54755)
0928 16:36:07.221633 (+ 1685us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:07.221696 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:07.222886 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:07.223745 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:07.223755 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:07.224158 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:07.225075 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.225097 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.226388 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.226391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.227896 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.227908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.228726 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.228730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.229027 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.229736 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:07.229760 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:07.231712 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:07.234151 (+ 2439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:07.234159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:07.234176 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:07.234536 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:07.234907 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:07.234910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:07.234912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:07.235098 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:07.235652 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:07.235658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:07.235991 (+ 333us) client_negotiation.cc:769] Sending connection context
0928 16:36:07.237366 (+ 1375us) client_negotiation.cc:240] Negotiation successful
0928 16:36:07.237746 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1436,"thread_start_us":150,"threads_started":1}
I20260928 16:36:07.240485 7334 heartbeater.cc:347] Connected to a master server at 127.27.122.254:35685
I20260928 16:36:07.240763 7334 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:07.241367 7334 heartbeater.cc:511] Master 127.27.122.254:35685 requested a full tablet report, sending...
I20260928 16:36:07.242560 6981 ts_manager.cc:194] Registered new tserver with Master: 4d89faa9a77d4a6098fb35c020b19ea6 (127.27.122.194:45217)
I20260928 16:36:07.243304 6981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:54755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:36:07.273284 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.122.212:42373/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35685
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:44633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:36:07.445425 7339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:07.445873 7339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:07.445986 7339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:07.451632 7339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:07.451768 7339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:07.451891 7339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:36:07.459245 7339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.122.212:42373/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:07.460696 7339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:07.462030 7339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:07.475315 7347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:07.475726 7345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:07.476649 7344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:07.476922 7339 server_base.cc:1034] running on GCE node
I20260928 16:36:07.477602 7339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:07.478375 7339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:07.479526 7339 hybrid_clock.cc:648] HybridClock initialized: now 1790613367479462 us; error 79 us; skew 500 ppm
Sep 28 16:36:07 dist-test-slave-g9sh krb5kdc[6501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613367, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:07.483003 7339 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:36:07.484012 7339 webserver.cc:460] Webserver started at http://127.27.122.195:37301/ using document root <none> and password file <none>
I20260928 16:36:07.484668 7339 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:07.484733 7339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:36:07.484997 7339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:36:07 dist-test-slave-g9sh krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613367, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:07.506341 7339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1"
format_stamp: "Formatted at 2026-09-28 16:36:07 on dist-test-slave-g9sh"
server_key: "93f6aff756a7d4a51115f059ed09c3ca"
server_key_iv: "215409cf3a447fd9078242ba944025d1"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:07.508005 7339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1"
format_stamp: "Formatted at 2026-09-28 16:36:07 on dist-test-slave-g9sh"
server_key: "93f6aff756a7d4a51115f059ed09c3ca"
server_key_iv: "215409cf3a447fd9078242ba944025d1"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:07.513224 7339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:36:07.538992 7355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:07.540732 7339 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:36:07.540851 7339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1"
format_stamp: "Formatted at 2026-09-28 16:36:07 on dist-test-slave-g9sh"
server_key: "93f6aff756a7d4a51115f059ed09c3ca"
server_key_iv: "215409cf3a447fd9078242ba944025d1"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:07.541005 7339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:07.567097 7339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:07.570717 7339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:07.570945 7339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:07.571601 7339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:36:07.572642 7339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:36:07.572738 7339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:07.572829 7339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:36:07.572888 7339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:07.587796 7469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:36359 every 8 connection(s)
I20260928 16:36:07.588004 7339 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:36359
I20260928 16:36:07.589157 7339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
Sep 28 16:36:07 dist-test-slave-g9sh krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613366, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:07.598798 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 7339
I20260928 16:36:07.598933 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:36:07.599260 28139 external_mini_cluster.cc:1428] Setting key b9dc85dd7c8dfe8f3b3fda73c723e9e0
Sep 28 16:36:07 dist-test-slave-g9sh krb5kdc[6501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613367, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:36:07.628350 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.593010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:34479 (local address 127.27.122.254:35685)
0928 16:36:07.593137 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:07.593142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:07.593563 (+ 421us) server_negotiation.cc:407] Connection header received
0928 16:36:07.594755 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:07.594758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:07.594814 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:07.594866 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:07.596462 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.597316 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.606493 (+ 9177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.606825 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.606930 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.611602 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:07.611606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:07.611609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:07.611636 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:07.613245 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:07.614151 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:07.614155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:07.614157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:07.614237 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:07.626700 (+ 12463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:07.626705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:07.626707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:07.626967 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:07.627060 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:07.628052 (+ 992us) server_negotiation.cc:299] Negotiation successful
0928 16:36:07.628188 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:36:07.630050 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.591948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35685 (local address 127.27.122.195:34479)
0928 16:36:07.593415 (+ 1467us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:07.593469 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:07.594513 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:07.595282 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:07.595291 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:07.595642 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:07.596294 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.596314 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.597824 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.597827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.606344 (+ 8517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.606357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.608245 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.608249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.608458 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.609199 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:07.609220 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:07.611413 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:07.613689 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:07.613694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:07.613712 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:07.614041 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:07.625104 (+ 11063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:07.625107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:07.625109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:07.625354 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:07.627131 (+ 1777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:07.627139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:07.627483 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 16:36:07.628568 (+ 1085us) client_negotiation.cc:240] Negotiation successful
0928 16:36:07.628976 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"thread_start_us":131,"threads_started":1}
I20260928 16:36:07.633462 7470 heartbeater.cc:347] Connected to a master server at 127.27.122.254:35685
I20260928 16:36:07.633700 7470 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:07.634258 7470 heartbeater.cc:511] Master 127.27.122.254:35685 requested a full tablet report, sending...
I20260928 16:36:07.636358 6981 ts_manager.cc:194] Registered new tserver with Master: ae8f52a061a04c9ea5f8c4a24aca8ec1 (127.27.122.195:36359)
I20260928 16:36:07.637085 6981 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:34479
I20260928 16:36:07.647423 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:36:07.663447 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.653414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.27.122.254:35685)
0928 16:36:07.653550 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:07.653554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:07.654131 (+ 577us) server_negotiation.cc:407] Connection header received
0928 16:36:07.654319 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:07.654322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:07.654372 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:07.654421 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:07.655421 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.656253 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.657517 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.657852 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.657928 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.659840 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:07.659844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:07.659847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:07.659872 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:07.661615 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:07.662113 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:07.662116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:07.662119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:07.662203 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:07.662506 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:07.662509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:07.662511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:07.662713 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:07.662812 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:07.663162 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 16:36:07.663279 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:36:07.667325 6981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:36:07.671783 6981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:36:07.705385 7482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.690415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:36359 (local address 127.0.0.1:33762)
0928 16:36:07.697184 (+ 6769us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:07.697198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:07.697287 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:07.697994 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:07.697997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:07.698016 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:36:07.698973 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.698982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.700581 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.700584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.701838 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.701847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.704804 (+ 2957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.704808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.704936 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.704993 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 16:36:07.705096 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 16:36:07.705220 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6707,"thread_start_us":101,"threads_started":1}
I20260928 16:36:07.705945 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.687668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.27.122.194:45217)
0928 16:36:07.700615 (+ 12947us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:07.700632 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:07.700672 (+ 40us) server_negotiation.cc:407] Connection header received
0928 16:36:07.700757 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:07.700770 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:07.700988 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:07.701169 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:36:07.701536 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.702673 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.704126 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.704883 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.705026 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.705158 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:07.705692 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 16:36:07.705797 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12833,"thread_start_us":68,"threads_started":1}
I20260928 16:36:07.706496 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.690827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.27.122.195:36359)
0928 16:36:07.692669 (+ 1842us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:07.692687 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:07.697388 (+ 4701us) server_negotiation.cc:407] Connection header received
0928 16:36:07.697487 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:07.697492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:07.697690 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:07.697839 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:36:07.699111 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.700409 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.702043 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.704500 (+ 2457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.704610 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.704741 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:07.706247 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0928 16:36:07.706363 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1732,"thread_start_us":134,"threads_started":1}
I20260928 16:36:07.705590 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.687555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:45217 (local address 127.0.0.1:37354)
0928 16:36:07.692177 (+ 4622us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:07.692228 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:07.692349 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:07.701314 (+ 8965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:07.701317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:07.701332 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:36:07.701415 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.701420 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.702835 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.702838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.703940 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.703951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.705216 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.705219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.705300 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.705319 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 16:36:07.705385 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 16:36:07.705460 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4439,"thread_start_us":107,"threads_started":1}
I20260928 16:36:07.706278 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.689364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:35309 (local address 127.0.0.1:33864)
0928 16:36:07.692824 (+ 3460us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:07.692837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:07.692950 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:07.693822 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:07.693828 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:07.693850 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:36:07.693973 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.693979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.696179 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.696183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.700318 (+ 4135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.700329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.705785 (+ 5456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.705788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.705874 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.705895 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:36:07.705947 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 16:36:07.706018 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3398,"thread_start_us":81,"threads_started":1}
I20260928 16:36:07.710091 7268 tablet_service.cc:1467] Processing CreateTablet for tablet 1890710440c84cdcab1c4797178c3a7f (DEFAULT_TABLE table=test-table [id=a33c9dfa9fa240848f3316b8b7ef4b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:36:07.710093 7403 tablet_service.cc:1467] Processing CreateTablet for tablet 1890710440c84cdcab1c4797178c3a7f (DEFAULT_TABLE table=test-table [id=a33c9dfa9fa240848f3316b8b7ef4b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:36:07.711232 7268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1890710440c84cdcab1c4797178c3a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:36:07.711232 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1890710440c84cdcab1c4797178c3a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:36:07.714308 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.689751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33864 (local address 127.27.122.193:35309)
0928 16:36:07.693182 (+ 3431us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:07.693189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:07.693213 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:36:07.693287 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:07.693293 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:07.693514 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:07.693653 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:36:07.694101 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.696046 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.703391 (+ 7345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.704252 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.706390 (+ 2138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.706525 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:07.706731 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 16:36:07.714164 (+ 7433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3327,"thread_start_us":89,"threads_started":1}
I20260928 16:36:07.717645 7129 tablet_service.cc:1467] Processing CreateTablet for tablet 1890710440c84cdcab1c4797178c3a7f (DEFAULT_TABLE table=test-table [id=a33c9dfa9fa240848f3316b8b7ef4b06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:36:07.718869 7129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1890710440c84cdcab1c4797178c3a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:36:07.720933 7489 tablet_bootstrap.cc:492] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1: Bootstrap starting.
I20260928 16:36:07.723877 7489 tablet_bootstrap.cc:654] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1: Neither blocks nor log segments found. Creating new log.
I20260928 16:36:07.724984 7489 log.cc:824] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1: Log is configured to *not* fsync() on all Append() calls
I20260928 16:36:07.727298 7489 tablet_bootstrap.cc:492] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1: No bootstrap required, opened a new log
I20260928 16:36:07.727488 7489 ts_tablet_manager.cc:1397] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:36:07.730371 7488 tablet_bootstrap.cc:492] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6: Bootstrap starting.
I20260928 16:36:07.731665 7489 raft_consensus.cc:348] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d89faa9a77d4a6098fb35c020b19ea6" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45217 } } peers { permanent_uuid: "af719e503f6545ebb5217ad1fc001aff" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35309 } } peers { permanent_uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 36359 } }
I20260928 16:36:07.732018 7489 raft_consensus.cc:374] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:36:07.732101 7489 raft_consensus.cc:729] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8f52a061a04c9ea5f8c4a24aca8ec1, State: Initialized, Role: FOLLOWER
I20260928 16:36:07.732620 7489 consensus_queue.cc:260] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d89faa9a77d4a6098fb35c020b19ea6" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45217 } } peers { permanent_uuid: "af719e503f6545ebb5217ad1fc001aff" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35309 } } peers { permanent_uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 36359 } }
I20260928 16:36:07.733592 7488 tablet_bootstrap.cc:654] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6: Neither blocks nor log segments found. Creating new log.
I20260928 16:36:07.733795 7489 ts_tablet_manager.cc:1428] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260928 16:36:07.734200 7470 heartbeater.cc:503] Master 127.27.122.254:35685 was elected leader, sending a full tablet report...
I20260928 16:36:07.734542 7488 log.cc:824] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:36:07.736604 7488 tablet_bootstrap.cc:492] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6: No bootstrap required, opened a new log
I20260928 16:36:07.736866 7488 ts_tablet_manager.cc:1397] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:36:07.738390 7491 tablet_bootstrap.cc:492] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff: Bootstrap starting.
I20260928 16:36:07.740748 7491 tablet_bootstrap.cc:654] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff: Neither blocks nor log segments found. Creating new log.
I20260928 16:36:07.741734 7491 log.cc:824] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff: Log is configured to *not* fsync() on all Append() calls
I20260928 16:36:07.741852 7488 raft_consensus.cc:348] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d89faa9a77d4a6098fb35c020b19ea6" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45217 } } peers { permanent_uuid: "af719e503f6545ebb5217ad1fc001aff" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35309 } } peers { permanent_uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 36359 } }
I20260928 16:36:07.742189 7488 raft_consensus.cc:374] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:36:07.742277 7488 raft_consensus.cc:729] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d89faa9a77d4a6098fb35c020b19ea6, State: Initialized, Role: FOLLOWER
I20260928 16:36:07.742820 7488 consensus_queue.cc:260] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d89faa9a77d4a6098fb35c020b19ea6" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45217 } } peers { permanent_uuid: "af719e503f6545ebb5217ad1fc001aff" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35309 } } peers { permanent_uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 36359 } }
I20260928 16:36:07.743775 7491 tablet_bootstrap.cc:492] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff: No bootstrap required, opened a new log
I20260928 16:36:07.743872 7488 ts_tablet_manager.cc:1428] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:36:07.743957 7491 ts_tablet_manager.cc:1397] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:36:07.744292 7334 heartbeater.cc:503] Master 127.27.122.254:35685 was elected leader, sending a full tablet report...
I20260928 16:36:07.749146 7491 raft_consensus.cc:348] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d89faa9a77d4a6098fb35c020b19ea6" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45217 } } peers { permanent_uuid: "af719e503f6545ebb5217ad1fc001aff" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35309 } } peers { permanent_uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 36359 } }
I20260928 16:36:07.749405 7491 raft_consensus.cc:374] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:36:07.749450 7491 raft_consensus.cc:729] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af719e503f6545ebb5217ad1fc001aff, State: Initialized, Role: FOLLOWER
I20260928 16:36:07.749928 7491 consensus_queue.cc:260] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d89faa9a77d4a6098fb35c020b19ea6" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45217 } } peers { permanent_uuid: "af719e503f6545ebb5217ad1fc001aff" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35309 } } peers { permanent_uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 36359 } }
I20260928 16:36:07.751008 7491 ts_tablet_manager.cc:1428] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:36:07.751324 7195 heartbeater.cc:503] Master 127.27.122.254:35685 was elected leader, sending a full tablet report...
I20260928 16:36:07.783808 7496 raft_consensus.cc:482] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:36:07.784076 7496 raft_consensus.cc:504] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d89faa9a77d4a6098fb35c020b19ea6" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45217 } } peers { permanent_uuid: "af719e503f6545ebb5217ad1fc001aff" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35309 } } peers { permanent_uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 36359 } }
I20260928 16:36:07.785521 7496 leader_election.cc:288] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d89faa9a77d4a6098fb35c020b19ea6 (127.27.122.194:45217), ae8f52a061a04c9ea5f8c4a24aca8ec1 (127.27.122.195:36359)
I20260928 16:36:07.790946 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.785888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:40281 (local address 127.27.122.194:45217)
0928 16:36:07.786039 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:07.786044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:07.786317 (+ 273us) server_negotiation.cc:407] Connection header received
0928 16:36:07.786568 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:07.786571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:07.786626 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:07.786674 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:36:07.787163 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.787950 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.789585 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.790197 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.790298 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.790361 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:07.790734 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 16:36:07.790813 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:36:07.791591 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.785768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:45217 (local address 127.27.122.193:40281)
0928 16:36:07.786224 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:07.786240 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:07.786401 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:07.786831 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:07.786834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:07.786853 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:36:07.787018 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.787025 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.788119 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.788123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.789436 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.789447 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.790473 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.790477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.790581 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.790608 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:36:07.791348 (+ 740us) client_negotiation.cc:240] Negotiation successful
0928 16:36:07.791429 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":116,"threads_started":1}
I20260928 16:36:07.792446 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1890710440c84cdcab1c4797178c3a7f" candidate_uuid: "af719e503f6545ebb5217ad1fc001aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d89faa9a77d4a6098fb35c020b19ea6" is_pre_election: true
I20260928 16:36:07.792680 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.786547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:46169 (local address 127.27.122.195:36359)
0928 16:36:07.786664 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:07.786669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:07.787946 (+ 1277us) server_negotiation.cc:407] Connection header received
0928 16:36:07.788186 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:07.788190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:07.788242 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:07.788293 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:36:07.788719 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.789528 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.791296 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.791919 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.792011 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.792058 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:07.792475 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 16:36:07.792564 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:36:07.792771 7288 raft_consensus.cc:2454] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af719e503f6545ebb5217ad1fc001aff in term 0.
I20260928 16:36:07.793123 7498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.786177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:36359 (local address 127.27.122.193:46169)
0928 16:36:07.787858 (+ 1681us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:07.787872 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:07.788014 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:07.788434 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:07.788436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:07.788454 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:36:07.788591 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.788598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.789696 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.789699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.791150 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:07.791161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.792167 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.792170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:07.792333 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.792355 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:36:07.792861 (+ 506us) client_negotiation.cc:240] Negotiation successful
0928 16:36:07.792956 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1615,"thread_start_us":92,"threads_started":1}
I20260928 16:36:07.793607 7084 leader_election.cc:302] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d89faa9a77d4a6098fb35c020b19ea6, af719e503f6545ebb5217ad1fc001aff; no voters:
I20260928 16:36:07.793828 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1890710440c84cdcab1c4797178c3a7f" candidate_uuid: "af719e503f6545ebb5217ad1fc001aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1" is_pre_election: true
I20260928 16:36:07.793912 7496 raft_consensus.cc:2781] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:36:07.793999 7496 raft_consensus.cc:482] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:36:07.794044 7496 raft_consensus.cc:3037] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:36:07.794113 7423 raft_consensus.cc:2454] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af719e503f6545ebb5217ad1fc001aff in term 0.
I20260928 16:36:07.795826 7496 raft_consensus.cc:504] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d89faa9a77d4a6098fb35c020b19ea6" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45217 } } peers { permanent_uuid: "af719e503f6545ebb5217ad1fc001aff" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35309 } } peers { permanent_uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 36359 } }
I20260928 16:36:07.796216 7496 leader_election.cc:288] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [CANDIDATE]: Term 1 election: Requested vote from peers 4d89faa9a77d4a6098fb35c020b19ea6 (127.27.122.194:45217), ae8f52a061a04c9ea5f8c4a24aca8ec1 (127.27.122.195:36359)
I20260928 16:36:07.796635 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1890710440c84cdcab1c4797178c3a7f" candidate_uuid: "af719e503f6545ebb5217ad1fc001aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1"
I20260928 16:36:07.796751 7423 raft_consensus.cc:3037] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:36:07.796756 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1890710440c84cdcab1c4797178c3a7f" candidate_uuid: "af719e503f6545ebb5217ad1fc001aff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d89faa9a77d4a6098fb35c020b19ea6"
I20260928 16:36:07.796952 7288 raft_consensus.cc:3037] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:36:07.798355 7423 raft_consensus.cc:2454] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af719e503f6545ebb5217ad1fc001aff in term 1.
I20260928 16:36:07.798595 7288 raft_consensus.cc:2454] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af719e503f6545ebb5217ad1fc001aff in term 1.
I20260928 16:36:07.798707 7084 leader_election.cc:302] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae8f52a061a04c9ea5f8c4a24aca8ec1, af719e503f6545ebb5217ad1fc001aff; no voters:
I20260928 16:36:07.798916 7496 raft_consensus.cc:2781] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:36:07.799227 7496 raft_consensus.cc:686] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [term 1 LEADER]: Becoming Leader. State: Replica: af719e503f6545ebb5217ad1fc001aff, State: Running, Role: LEADER
I20260928 16:36:07.799607 7496 consensus_queue.cc:237] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d89faa9a77d4a6098fb35c020b19ea6" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45217 } } peers { permanent_uuid: "af719e503f6545ebb5217ad1fc001aff" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35309 } } peers { permanent_uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 36359 } }
I20260928 16:36:07.803503 6980 catalog_manager.cc:5495] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff reported cstate change: term changed from 0 to 1, leader changed from <none> to af719e503f6545ebb5217ad1fc001aff (127.27.122.193). New cstate: current_term: 1 leader_uuid: "af719e503f6545ebb5217ad1fc001aff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d89faa9a77d4a6098fb35c020b19ea6" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af719e503f6545ebb5217ad1fc001aff" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 36359 } health_report { overall_health: UNKNOWN } } }
W20260928 16:36:07.840278 7471 tablet.cc:2369] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:36:07.849797 7485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.845830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33874 (local address 127.27.122.193:35309)
0928 16:36:07.845948 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:07.845953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:07.845994 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:36:07.846203 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:07.846206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:07.846262 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:07.846372 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:36:07.846617 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.847200 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.848290 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:07.848556 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:07.848670 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:07.849007 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:36:07.849419 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:36:07.849569 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:07.849610 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 16:36:07.849665 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 16:36:07.851455 7196 tablet.cc:2369] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:36:07.856309 7423 raft_consensus.cc:1261] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1 [term 1 FOLLOWER]: Refusing update from remote peer af719e503f6545ebb5217ad1fc001aff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:36:07.857164 7499 consensus_queue.cc:1035] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8f52a061a04c9ea5f8c4a24aca8ec1" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 36359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:36:07.858045 7288 raft_consensus.cc:1261] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6 [term 1 FOLLOWER]: Refusing update from remote peer af719e503f6545ebb5217ad1fc001aff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:36:07.859521 7499 consensus_queue.cc:1035] T 1890710440c84cdcab1c4797178c3a7f P af719e503f6545ebb5217ad1fc001aff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d89faa9a77d4a6098fb35c020b19ea6" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:36:07.879679 7503 mvcc.cc:205] Tried to move back new op lower bound from 7334352354731888640 to 7334352354510659584. Current Snapshot: MvccSnapshot[applied={T|T < 7334352354731888640}]
I20260928 16:36:07.880206 7505 mvcc.cc:205] Tried to move back new op lower bound from 7334352354731888640 to 7334352354510659584. Current Snapshot: MvccSnapshot[applied={T|T < 7334352354731888640}]
I20260928 16:36:07.881304 6980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:36:07.881495 6980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:36:07.884431 6980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1810dccc3afd4a02add0ce6a8bd3035b: Sending DeleteTablet for 3 replicas of tablet 1890710440c84cdcab1c4797178c3a7f
I20260928 16:36:07.885524 7129 tablet_service.cc:1514] Processing DeleteTablet for tablet 1890710440c84cdcab1c4797178c3a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:36:07 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:33864
I20260928 16:36:07.886749 7268 tablet_service.cc:1514] Processing DeleteTablet for tablet 1890710440c84cdcab1c4797178c3a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:36:07 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:37354
I20260928 16:36:07.887334 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet 1890710440c84cdcab1c4797178c3a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:36:07 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:33762
I20260928 16:36:07.889067 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 7063
I20260928 16:36:07.890933 7513 tablet_replica.cc:330] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1: stopping tablet replica
I20260928 16:36:07.891000 7512 tablet_replica.cc:330] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6: stopping tablet replica
I20260928 16:36:07.891307 7513 raft_consensus.cc:2229] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:36:07.891331 7512 raft_consensus.cc:2229] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:36:07.891644 7512 raft_consensus.cc:2258] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:36:07.891644 7513 raft_consensus.cc:2258] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:36:07.893744 7512 ts_tablet_manager.cc:1905] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:36:07.893744 7513 ts_tablet_manager.cc:1905] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:36:07.896668 7512 ts_tablet_manager.cc:1918] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:36:07.896668 7513 ts_tablet_manager.cc:1918] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:36:07.896776 7513 log.cc:1196] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/1890710440c84cdcab1c4797178c3a7f
I20260928 16:36:07.896926 7512 log.cc:1196] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/1890710440c84cdcab1c4797178c3a7f
I20260928 16:36:07.897219 7513 ts_tablet_manager.cc:1939] T 1890710440c84cdcab1c4797178c3a7f P ae8f52a061a04c9ea5f8c4a24aca8ec1: Deleting consensus metadata
I20260928 16:36:07.897331 7512 ts_tablet_manager.cc:1939] T 1890710440c84cdcab1c4797178c3a7f P 4d89faa9a77d4a6098fb35c020b19ea6: Deleting consensus metadata
I20260928 16:36:07.898778 6968 catalog_manager.cc:4841] TS 4d89faa9a77d4a6098fb35c020b19ea6 (127.27.122.194:45217): tablet 1890710440c84cdcab1c4797178c3a7f (table test-table [id=a33c9dfa9fa240848f3316b8b7ef4b06]) successfully deleted
I20260928 16:36:07.902267 6968 catalog_manager.cc:4841] TS ae8f52a061a04c9ea5f8c4a24aca8ec1 (127.27.122.195:36359): tablet 1890710440c84cdcab1c4797178c3a7f (table test-table [id=a33c9dfa9fa240848f3316b8b7ef4b06]) successfully deleted
W20260928 16:36:07.902560 6966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:35309 (error 108)
I20260928 16:36:07.903479 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 7203
I20260928 16:36:07.903746 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:07.903306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:35309 (local address 127.0.0.1:33878)
0928 16:36:07.903551 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:07.903637 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:35309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260928 16:36:07.904350 6966 catalog_manager.cc:4584] TS af719e503f6545ebb5217ad1fc001aff (127.27.122.193:35309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1890710440c84cdcab1c4797178c3a7f: Network error: Client connection negotiation failed: client connection to 127.27.122.193:35309: connect: Connection refused (error 111)
I20260928 16:36:07.911830 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 7339
I20260928 16:36:07.920233 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 6949
I20260928 16:36:07.928398 28139 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 16:36:08.272825 28139 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 16:36:08.273059 28139 mini_ranger.cc:66] Stopping Ranger...
I20260928 16:36:08.617769 28139 mini_ranger.cc:68] Stopped Ranger
2026-09-28 16:36:08.618 UTC [6531] LOG: received smart shutdown request
2026-09-28 16:36:08.622 UTC [6531] LOG: background worker "logical replication launcher" (PID 6540) exited with exit code 1
2026-09-28 16:36:08.622 UTC [6535] LOG: shutting down
2026-09-28 16:36:08.793 UTC [6531] LOG: database system is shut down
2026-09-28T16:36:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49415 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:36:08 dist-test-slave-g9sh krb5kdc[7523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:36:08 dist-test-slave-g9sh krb5kdc[7523](info): set up 2 sockets
Sep 28 16:36:08 dist-test-slave-g9sh krb5kdc[7523](info): commencing operation
krb5kdc: starting...
W20260928 16:36:09.956475 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:36:10 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:36:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:36:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.122.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.122.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/rangeradmin_127.27.122.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/rangeradmin_127.27.122.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.122.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.122.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/rangerlookup_127.27.122.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/rangerlookup_127.27.122.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.122.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.122.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/HTTP_127.27.122.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/HTTP_127.27.122.212@KRBTEST.COM.keytab.
I20260928 16:36:10.119145 28139 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYGQpme/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/postgres -l logfile start
2026-09-28 16:36:12.538 UTC [7555] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 16:36:12.538 UTC [7555] LOG: listening on IPv4 address "127.27.122.212", port 38309
2026-09-28 16:36:12.541 UTC [7555] LOG: listening on Unix socket "/tmp/.s.PGSQL.38309"
2026-09-28 16:36:12.562 UTC [7559] LOG: database system was shut down at 2026-09-28 16:36:11 UTC
2026-09-28 16:36:12.565 UTC [7555] LOG: database system is ready to accept connections
I20260928 16:36:13.523254 28139 mini_postgres.cc:96] Postgres bound to 38309
2026-09-28 16:36:13.529 UTC [7568] FATAL: database "slave" does not exist
127.27.122.212:38309 - accepting connections
I20260928 16:36:13.530169 28139 mini_ranger.cc:161] Starting Ranger...
I20260928 16:36:13.545739 28139 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 16:36:13.970404 28139 mini_ranger.cc:87] Created ranger Postgres database
I20260928 16:36:13.970495 28139 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-admin
2026-09-28 16:36:14,245 [I] DB FLAVOR :POSTGRES
2026-09-28 16:36:14,245 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:36:14,245 [I] Checking connection..
2026-09-28 16:36:14,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 16:36:14,506 [I] Checking connection passed.
2026-09-28 16:36:14,506 [I] --------- Verifying version history table ---------
2026-09-28 16:36:14,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:36:14,766 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:36:14,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:36:15,027 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 16:36:15,027 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 16:36:15,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 16:36:15.260 UTC [7633] WARNING: there is no transaction in progress
2026-09-28 16:36:15,266 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 16:36:15,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 16:36:15,529 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 16:36:15,529 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:36:15,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 16:36:15,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 16:36:16,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g9sh.c.gcp-upstream.internal','N') ;"
2026-09-28 16:36:16,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:36:16,471 [I] Table x_portal_user does not exist in database ranger
2026-09-28 16:36:16,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:36:16,723 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 16:36:16,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:36:16,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:36:17,229 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 16:36:17,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 16:36:18.157 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.173 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.189 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.203 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.218 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.235 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.288 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.295 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.309 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.319 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.329 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.342 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.352 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.358 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.365 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18.371 UTC [7743] WARNING: there is no transaction in progress
2026-09-28 16:36:18,985 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 16:36:18,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 16:36:19,238 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 16:36:19,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 16:36:19,498 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 16:36:19,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 16:36:19,759 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 16:36:19,759 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 16:36:20,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 16:36:20,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g9sh.c.gcp-upstream.internal';"
2026-09-28 16:36:20,524 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 16:36:20,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 16:36:20,781 [I] DB_PATCHES have already been applied
I20260928 16:36:20.787642 28139 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYGQpme/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 16:36:20.787704 28139 mini_ranger.cc:193] Using host: 127.27.122.212
I20260928 16:36:20.791260 28139 mini_ranger.cc:227] Ranger admin URL: http://127.27.122.212:56183
Sep 28, 2026 4:36:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:36:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:36:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 4:36:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 4:36:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 4:36:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 4:36:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 16:36:21.908941 28139 mini_ranger.cc:160] Time spent starting Ranger: real 8.379s user 0.002s sys 0.007s
Sep 28, 2026 4:36:41 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 4:36:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 4:36:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 4:36:41 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 4:36:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 4:36:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 4:36:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 4:36:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 4:36:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 4:36:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 4:36:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 4:36:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 4:36:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 4:36:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 4:36:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 4:36:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 4:36:41 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 4:36:42 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 16:36:42.702047 28139 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.122.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.122.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/rangerkms_127.27.122.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.122.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/rangerkms_127.27.122.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.122.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.122.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.122.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/HTTP_127.27.122.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.122.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/HTTP_127.27.122.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 16:36:42 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613402, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 16:36:42.790311 28139 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 16:36:42.799539 28139 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 16:36:43.571008 28139 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 16:36:43.571099 28139 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-kms
I20260928 16:36:43.571121 28139 mini_ranger_kms.cc:227] Using postgres at 127.27.122.212:38309
2026-09-28 16:36:43,746 [I] DB FLAVOR :POSTGRES
2026-09-28 16:36:43,746 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 16:36:43,746 [I] Checking connection
2026-09-28 16:36:43,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 16:36:43,976 [I] connection success
2026-09-28 16:36:43,976 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 16:36:43,976 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 16:36:43,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 16:36:44,223 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 16:36:44,224 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 16:36:44,224 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 16:36:44,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.122.212:38309/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 16:36:44,478 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 16:36:44.900130 28139 mini_ranger_kms.cc:310] Created kms service
I20260928 16:36:45.119535 28139 mini_ranger_kms.cc:326] Created kudu user
I20260928 16:36:45.162789 28139 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 16:36:45 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613402, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:45.554072 28139 mini_ranger_kms.cc:384] Added ranger policy
I20260928 16:36:45.554304 28139 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYGQpme/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYGQpme/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 16:36:45.554337 28139 mini_ranger_kms.cc:240] Using host: 127.27.122.212
I20260928 16:36:45.558228 28139 mini_ranger_kms.cc:276] Ranger KMS PID: 7927
I20260928 16:36:45.558314 28139 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.122.212:39261
Sep 28, 2026 4:36:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 4:36:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 4:36:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 4:36:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 4:36:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 4:36:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 16:36:47.828147 28139 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.038s user 0.015s sys 0.144s
I20260928 16:36:47.828276 28139 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 16:36:47.828358 28139 mini_ranger_kms.cc:401] 127.27.122.212:39261/kms/v1/keys
Sep 28 16:36:47 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613407, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.122.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 16:36:48 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613407, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.122.212@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
Sep 28 16:36:48 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613407, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.122.212@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
Sep 28 16:36:50 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:36:50.904292 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:37581
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:37581
--ranger_config_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.122.212:39261/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:36:51.012281 7978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:51.012514 7978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:51.012550 7978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:51.016461 7978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:36:51.016511 7978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:51.016556 7978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:51.016592 7978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:36:51.016629 7978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:36:51.021844 7978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.122.212:39261/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:37581
--ranger_config_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:37581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:51.022768 7978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:51.023717 7978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:51.032397 7984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:51.032426 7986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:51.032550 7978 server_base.cc:1034] running on GCE node
W20260928 16:36:51.032428 7983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:51.033269 7978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:51.034384 7978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:51.037650 7978 hybrid_clock.cc:648] HybridClock initialized: now 1790613411037606 us; error 52 us; skew 500 ppm
Sep 28 16:36:51 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613411, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:51.040244 7978 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:36:51.041144 7978 webserver.cc:460] Webserver started at http://127.27.122.254:42515/ using document root <none> and password file <none>
I20260928 16:36:51.041664 7978 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:51.041740 7978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:36:51.041958 7978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:36:51 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613411, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:51.094070 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "b9a578998001409a8c2349d0637b8358"
format_stamp: "Formatted at 2026-09-28 16:36:51 on dist-test-slave-g9sh"
server_key: "5e62de3c4e908eb03da96f9453ff838d"
server_key_iv: "cad3b7d53c5762b505ba2c2d39fe1c78"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:51.094692 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "b9a578998001409a8c2349d0637b8358"
format_stamp: "Formatted at 2026-09-28 16:36:51 on dist-test-slave-g9sh"
server_key: "5e62de3c4e908eb03da96f9453ff838d"
server_key_iv: "cad3b7d53c5762b505ba2c2d39fe1c78"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:51.098678 7978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 16:36:51.115343 7993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:51.116695 7978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 16:36:51.116824 7978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "b9a578998001409a8c2349d0637b8358"
format_stamp: "Formatted at 2026-09-28 16:36:51 on dist-test-slave-g9sh"
server_key: "5e62de3c4e908eb03da96f9453ff838d"
server_key_iv: "cad3b7d53c5762b505ba2c2d39fe1c78"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:51.116992 7978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:51.129948 7978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:51.132737 7978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:51.132952 7978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:51.140930 7978 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:37581
I20260928 16:36:51.140982 8045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:37581 every 8 connection(s)
I20260928 16:36:51.141968 7978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:36:51.145421 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:36:51.150517 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 7978
I20260928 16:36:51.150657 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:36:51.150965 28139 external_mini_cluster.cc:1428] Setting key 7448f41664baa49a178345be79d5a9a7
I20260928 16:36:51.152462 8046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358: Bootstrap starting.
I20260928 16:36:51.155324 8046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358: Neither blocks nor log segments found. Creating new log.
I20260928 16:36:51.156208 8046 log.cc:824] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358: Log is configured to *not* fsync() on all Append() calls
I20260928 16:36:51.158414 8046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358: No bootstrap required, opened a new log
Sep 28 16:36:51 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:36:51.162801 8046 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a578998001409a8c2349d0637b8358" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37581 } }
I20260928 16:36:51.163070 8046 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:36:51.163157 8046 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a578998001409a8c2349d0637b8358, State: Initialized, Role: FOLLOWER
I20260928 16:36:51.163651 8046 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a578998001409a8c2349d0637b8358" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37581 } }
I20260928 16:36:51.163816 8046 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:36:51.163909 8046 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:36:51.164074 8046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:36:51.165058 8046 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a578998001409a8c2349d0637b8358" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37581 } }
I20260928 16:36:51.165144 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:51.152398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.27.122.254:37581)
0928 16:36:51.152933 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:51.152953 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:51.152995 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:36:51.153675 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:51.153694 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:51.154019 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:51.154354 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:51.155150 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:51.156180 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:51.157248 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:51.157714 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:51.157890 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:51.160355 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:51.160360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:51.160373 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:51.160398 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:51.162055 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:51.162499 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:51.162504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:51.162508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:51.162608 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:51.162892 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:51.162897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:51.162899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:51.163376 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:51.163571 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:51.164079 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 16:36:51.164365 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":149,"threads_started":1}
I20260928 16:36:51.165413 8046 leader_election.cc:302] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9a578998001409a8c2349d0637b8358; no voters:
I20260928 16:36:51.165746 8046 leader_election.cc:288] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:36:51.165848 8051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:36:51.166013 8051 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 1 LEADER]: Becoming Leader. State: Replica: b9a578998001409a8c2349d0637b8358, State: Running, Role: LEADER
I20260928 16:36:51.166358 8051 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a578998001409a8c2349d0637b8358" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37581 } }
I20260928 16:36:51.166989 8046 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:36:51.167488 8051 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9a578998001409a8c2349d0637b8358. Latest consensus state: current_term: 1 leader_uuid: "b9a578998001409a8c2349d0637b8358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a578998001409a8c2349d0637b8358" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37581 } } }
I20260928 16:36:51.167599 8051 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [sys.catalog]: This master's current role is: LEADER
I20260928 16:36:51.167873 8052 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9a578998001409a8c2349d0637b8358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a578998001409a8c2349d0637b8358" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37581 } } }
I20260928 16:36:51.168026 8052 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [sys.catalog]: This master's current role is: LEADER
I20260928 16:36:51.169099 8056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:36:51.169325 8046 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:36:51.174660 8056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:36:51.181334 8056 catalog_manager.cc:1347] Generated new cluster ID: 56e5af54b54749f2a8242e30c87f6440
I20260928 16:36:51.181396 8056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:36:51.206440 8056 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:36:51.207760 8056 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:36:51.226775 8056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358: Generated new TSK 0
I20260928 16:36:51.227751 8056 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:36:51.854106 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.122.212:39261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:37581
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:36:51.993181 8093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:51.993458 8093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:51.993582 8093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:51.998930 8093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:51.999009 8093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:51.999094 8093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:36:52.006675 8093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.122.212:39261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:37581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:52.007966 8093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:52.009255 8093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:52.019346 8100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:52.019327 8103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:52.019922 8093 server_base.cc:1034] running on GCE node
W20260928 16:36:52.020018 8101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:52.020411 8093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:52.021185 8093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:52.022351 8093 hybrid_clock.cc:648] HybridClock initialized: now 1790613412022322 us; error 50 us; skew 500 ppm
Sep 28 16:36:52 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:52.025202 8093 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:36:52.026124 8093 webserver.cc:460] Webserver started at http://127.27.122.193:38131/ using document root <none> and password file <none>
I20260928 16:36:52.026705 8093 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:52.026793 8093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:36:52.027066 8093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:36:52 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:52.046864 8093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "f2978040a6204764b4ae2d740b68b353"
format_stamp: "Formatted at 2026-09-28 16:36:52 on dist-test-slave-g9sh"
server_key: "c7e5eef41c46be91967d8d3e3d8559ba"
server_key_iv: "4febc18fa2977dc29475a768d4d2ed9b"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:52.048681 8093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "f2978040a6204764b4ae2d740b68b353"
format_stamp: "Formatted at 2026-09-28 16:36:52 on dist-test-slave-g9sh"
server_key: "c7e5eef41c46be91967d8d3e3d8559ba"
server_key_iv: "4febc18fa2977dc29475a768d4d2ed9b"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:52.056159 8093 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.003s
I20260928 16:36:52.081354 8112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:52.083135 8093 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.001s
I20260928 16:36:52.083328 8093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "f2978040a6204764b4ae2d740b68b353"
format_stamp: "Formatted at 2026-09-28 16:36:52 on dist-test-slave-g9sh"
server_key: "c7e5eef41c46be91967d8d3e3d8559ba"
server_key_iv: "4febc18fa2977dc29475a768d4d2ed9b"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:52.083559 8093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:52.105548 8093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:52.108503 8093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:52.108695 8093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:52.109350 8093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:36:52.110237 8093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:36:52.110303 8093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:52.110409 8093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:36:52.110458 8093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:52.121730 8093 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:34627
I20260928 16:36:52.121760 8227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:34627 every 8 connection(s)
I20260928 16:36:52.122747 8093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:36:52.131834 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 8093
I20260928 16:36:52.131971 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:36:52.132261 28139 external_mini_cluster.cc:1428] Setting key edcfc4de366c94bbbc57a71417af7390
Sep 28 16:36:52 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:36:52.147979 8231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:52.125254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37581 (local address 127.27.122.193:33153)
0928 16:36:52.125718 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:52.125770 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:52.126716 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:52.127553 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:52.127565 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:52.127929 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:52.128601 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:52.128631 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:52.129871 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:52.129875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:52.131038 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:52.131049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:52.131547 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:52.131551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:52.132828 (+ 1277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:52.133531 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:52.133554 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:52.136836 (+ 3282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:52.139498 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:52.139507 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:52.139527 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:52.139923 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:52.140310 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:52.140317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:52.140321 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:52.145576 (+ 5255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:52.146106 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:52.146113 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:52.146418 (+ 305us) client_negotiation.cc:769] Sending connection context
0928 16:36:52.146610 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 16:36:52.146964 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":129,"threads_started":1}
I20260928 16:36:52.148806 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:52.124725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:33153 (local address 127.27.122.254:37581)
0928 16:36:52.125034 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:52.125042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:52.125869 (+ 827us) server_negotiation.cc:407] Connection header received
0928 16:36:52.126969 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:52.126973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:52.127038 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:52.127110 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:52.128798 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:52.129706 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:52.131170 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:52.131399 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:52.131508 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:52.137141 (+ 5633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:52.137147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:52.137153 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:52.137186 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:52.139358 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:52.140059 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:52.140065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:52.140070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:52.140188 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:52.145715 (+ 5527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:52.145719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:52.145720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:52.145954 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:52.146051 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:52.148495 (+ 2444us) server_negotiation.cc:299] Negotiation successful
0928 16:36:52.148636 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
I20260928 16:36:52.150488 8228 heartbeater.cc:347] Connected to a master server at 127.27.122.254:37581
I20260928 16:36:52.150733 8228 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:52.151288 8228 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
I20260928 16:36:52.153012 8010 ts_manager.cc:194] Registered new tserver with Master: f2978040a6204764b4ae2d740b68b353 (127.27.122.193:34627)
I20260928 16:36:52.154448 8010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:33153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
Sep 28 16:36:52 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:36:52.176405 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.122.212:39261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:37581
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:36:52.332942 8234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:52.333283 8234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:52.333350 8234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:52.337999 8234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:52.338055 8234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:52.338150 8234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:36:52.343668 8234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.122.212:39261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:37581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:52.344727 8234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:52.345981 8234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:52.357335 8241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:52.357652 8243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:52.357785 8240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:52.359424 8234 server_base.cc:1034] running on GCE node
I20260928 16:36:52.359846 8234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:52.360457 8234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:52.361785 8234 hybrid_clock.cc:648] HybridClock initialized: now 1790613412361751 us; error 45 us; skew 500 ppm
Sep 28 16:36:52 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:52.364854 8234 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:36:52.365934 8234 webserver.cc:460] Webserver started at http://127.27.122.194:41575/ using document root <none> and password file <none>
I20260928 16:36:52.366619 8234 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:52.366734 8234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:36:52.366972 8234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:36:52 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:52.383069 8234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "e30e45d0347f4146a6bf5bad220282bd"
format_stamp: "Formatted at 2026-09-28 16:36:52 on dist-test-slave-g9sh"
server_key: "da23bfafc1a692f24d7f6c05d6881308"
server_key_iv: "2cc16fb08672feb87588174b50d08d03"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:52.383828 8234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "e30e45d0347f4146a6bf5bad220282bd"
format_stamp: "Formatted at 2026-09-28 16:36:52 on dist-test-slave-g9sh"
server_key: "da23bfafc1a692f24d7f6c05d6881308"
server_key_iv: "2cc16fb08672feb87588174b50d08d03"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:52.388007 8234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 16:36:52.409168 8250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:52.410933 8234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:36:52.411079 8234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "e30e45d0347f4146a6bf5bad220282bd"
format_stamp: "Formatted at 2026-09-28 16:36:52 on dist-test-slave-g9sh"
server_key: "da23bfafc1a692f24d7f6c05d6881308"
server_key_iv: "2cc16fb08672feb87588174b50d08d03"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:52.411234 8234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:52.435326 8234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:52.439198 8234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:52.439435 8234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:52.440200 8234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:36:52.441304 8234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:36:52.441413 8234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:52.441529 8234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:36:52.441598 8234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:52.454380 8234 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:38683
I20260928 16:36:52.455035 8363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:38683 every 8 connection(s)
I20260928 16:36:52.455490 8234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:36:52.460788 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 8234
I20260928 16:36:52.460940 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:36:52.461272 28139 external_mini_cluster.cc:1428] Setting key f0099585eb8cb8d86755462ffca23922
Sep 28 16:36:52 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:36:52.472937 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:52.458848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37581 (local address 127.27.122.194:37381)
0928 16:36:52.459379 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:52.459421 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:52.460296 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:52.460919 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:52.460930 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:52.461220 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:52.461808 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:52.461829 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:52.462850 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:52.462853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:52.464247 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:52.464263 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:52.465082 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:52.465087 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:52.465275 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:52.465914 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:52.465930 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:52.467450 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:52.469680 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:52.469688 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:52.469703 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:52.469956 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:52.470269 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:52.470274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:52.470278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:52.470459 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:52.470958 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:52.470965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:52.471301 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 16:36:52.471519 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 16:36:52.471862 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":151,"threads_started":1}
I20260928 16:36:52.473651 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:52.458337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:37381 (local address 127.27.122.254:37581)
0928 16:36:52.458462 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:52.458466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:52.459528 (+ 1062us) server_negotiation.cc:407] Connection header received
0928 16:36:52.460567 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:52.460570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:52.460628 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:52.460728 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:52.461960 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:52.462699 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:52.464392 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:52.464787 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:52.464865 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:52.467619 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:52.467625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:52.467629 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:52.467654 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:52.469579 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:52.470056 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:52.470062 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:52.470065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:52.470156 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:52.470571 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:52.470574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:52.470577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:52.470801 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:52.470898 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:52.473353 (+ 2455us) server_negotiation.cc:299] Negotiation successful
0928 16:36:52.473481 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
I20260928 16:36:52.477151 8364 heartbeater.cc:347] Connected to a master server at 127.27.122.254:37581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
I20260928 16:36:52.477397 8364 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:52.477952 8364 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
I20260928 16:36:52.479043 8010 ts_manager.cc:194] Registered new tserver with Master: e30e45d0347f4146a6bf5bad220282bd (127.27.122.194:38683)
I20260928 16:36:52.479892 8010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:37381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:36:52.495491 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.122.212:39261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:37581
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:36:52 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
W20260928 16:36:52.638757 8370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:52.639039 8370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:52.639174 8370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:52.643386 8370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:52.643458 8370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:52.643573 8370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:36:52.648991 8370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.122.212:39261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:37581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:52.650071 8370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:52.651119 8370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:52.661185 8376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:52.661386 8370 server_base.cc:1034] running on GCE node
W20260928 16:36:52.661281 8378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:52.661334 8375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:52.661980 8370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:52.662853 8370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:52.664086 8370 hybrid_clock.cc:648] HybridClock initialized: now 1790613412664051 us; error 47 us; skew 500 ppm
Sep 28 16:36:52 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:52.666867 8370 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:36:52.667807 8370 webserver.cc:460] Webserver started at http://127.27.122.195:38679/ using document root <none> and password file <none>
I20260928 16:36:52.668524 8370 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:52.668615 8370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:36:52.668848 8370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 16:36:52 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:52.693575 8370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "916c4cbac84249ffa0eba0962517f0b6"
format_stamp: "Formatted at 2026-09-28 16:36:52 on dist-test-slave-g9sh"
server_key: "c493fe3ead742afd8002a12a55b4155d"
server_key_iv: "141fee043358bbd7ebcccaba9d79a573"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:52.694262 8370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "916c4cbac84249ffa0eba0962517f0b6"
format_stamp: "Formatted at 2026-09-28 16:36:52 on dist-test-slave-g9sh"
server_key: "c493fe3ead742afd8002a12a55b4155d"
server_key_iv: "141fee043358bbd7ebcccaba9d79a573"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:52.699527 8370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 16:36:52.728026 8386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:52.729909 8370 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260928 16:36:52.730063 8370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "916c4cbac84249ffa0eba0962517f0b6"
format_stamp: "Formatted at 2026-09-28 16:36:52 on dist-test-slave-g9sh"
server_key: "c493fe3ead742afd8002a12a55b4155d"
server_key_iv: "141fee043358bbd7ebcccaba9d79a573"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:52.730207 8370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:52.754782 8370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:52.758366 8370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:52.758544 8370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:52.759240 8370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:36:52.760249 8370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:36:52.760311 8370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:52.760363 8370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:36:52.760437 8370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:52.771026 8370 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:42187
I20260928 16:36:52.771104 8502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:42187 every 8 connection(s)
I20260928 16:36:52.772146 8370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:36:52.774001 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 8370
I20260928 16:36:52.774129 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:36:52.774449 28139 external_mini_cluster.cc:1428] Setting key eeb9d414875e00d7aa288b007f9e3f77
Sep 28 16:36:52 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:36:52.788268 8230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:52.775894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:33285 (local address 127.27.122.254:37581)
0928 16:36:52.776441 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:52.776446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:52.776460 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:36:52.776596 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:52.776599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:52.776652 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:52.776735 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:52.778275 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:52.779050 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:52.780659 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:52.781039 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:52.781129 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:52.784028 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:52.784032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:52.784035 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:52.784061 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:52.785724 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:52.786390 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:52.786394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:52.786396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:52.786486 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:52.786952 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:52.786955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:52.786957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:52.787193 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:52.787280 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:52.787994 (+ 714us) server_negotiation.cc:299] Negotiation successful
0928 16:36:52.788120 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477}
I20260928 16:36:52.790059 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:52.774730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37581 (local address 127.27.122.195:33285)
0928 16:36:52.775278 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:52.775339 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:52.776339 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:52.777089 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:52.777100 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:52.777459 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:52.778114 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:52.778134 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:52.779218 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:52.779221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:52.780524 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:52.780535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:52.781326 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:52.781330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:52.781510 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:52.782164 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:52.782184 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:52.783847 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:52.785852 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:52.785858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:52.785872 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:52.786224 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:52.786593 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:52.786597 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:52.786599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:52.786794 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:52.787389 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:52.787395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:52.787709 (+ 314us) client_negotiation.cc:769] Sending connection context
0928 16:36:52.788495 (+ 786us) client_negotiation.cc:240] Negotiation successful
0928 16:36:52.788933 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":161,"threads_started":1}
I20260928 16:36:52.791419 8503 heartbeater.cc:347] Connected to a master server at 127.27.122.254:37581
I20260928 16:36:52.791641 8503 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:52.792168 8503 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
I20260928 16:36:52.793264 8010 ts_manager.cc:194] Registered new tserver with Master: 916c4cbac84249ffa0eba0962517f0b6 (127.27.122.195:42187)
I20260928 16:36:52.793988 8010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:33285
I20260928 16:36:52.800114 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:36:52.801100 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 8093
I20260928 16:36:52.809297 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 8234
I20260928 16:36:52.816233 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 8370
I20260928 16:36:52.821987 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 7978
I20260928 16:36:52.828409 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:37581
--webserver_interface=127.27.122.254
--webserver_port=42515
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:37581
--ranger_config_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.122.212:39261/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.122.212:39261/kms with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:36:52.936889 8506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:52.937150 8506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:52.937187 8506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:52.941174 8506 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:36:52.941223 8506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:52.941243 8506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:52.941306 8506 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:36:52.941341 8506 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:36:52.946692 8506 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.122.212:39261/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:37581
--ranger_config_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:37581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=42515
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:52.947721 8506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:52.948714 8506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:52.957304 8514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:52.957337 8512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:52.957321 8511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:52.957639 8506 server_base.cc:1034] running on GCE node
I20260928 16:36:52.958225 8506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:52.959255 8506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:52.960441 8506 hybrid_clock.cc:648] HybridClock initialized: now 1790613412960430 us; error 33 us; skew 500 ppm
Sep 28 16:36:52 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:52.963099 8506 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:36:52.963972 8506 webserver.cc:460] Webserver started at http://127.27.122.254:42515/ using document root <none> and password file <none>
I20260928 16:36:52.964486 8506 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:52.964583 8506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:36:53 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613412, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:53.024695 8506 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:36:53.026937 8522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:53.028270 8506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:36:53.028410 8506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "b9a578998001409a8c2349d0637b8358"
format_stamp: "Formatted at 2026-09-28 16:36:51 on dist-test-slave-g9sh"
server_key: "5e62de3c4e908eb03da96f9453ff838d"
server_key_iv: "cad3b7d53c5762b505ba2c2d39fe1c78"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:53.029023 8506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:53.050416 8506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:53.052981 8506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:53.053181 8506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:53.060530 8506 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:37581
I20260928 16:36:53.060551 8576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:37581 every 8 connection(s)
I20260928 16:36:53.061631 8506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:36:53.065183 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 8506
I20260928 16:36:53.065621 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.122.212:39261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:34627
--local_ip_for_outbound_sockets=127.27.122.193
--tserver_master_addrs=127.27.122.254:37581
--webserver_port=38131
--webserver_interface=127.27.122.193
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.122.212:39261/kms with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:36:53.065943 8577 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:36:53.067521 8577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358: Bootstrap starting.
I20260928 16:36:53.083894 8577 log.cc:824] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358: Log is configured to *not* fsync() on all Append() calls
I20260928 16:36:53.090404 8577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:36:53.090796 8577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358: Bootstrap complete.
I20260928 16:36:53.094933 8577 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a578998001409a8c2349d0637b8358" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37581 } }
I20260928 16:36:53.095345 8577 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9a578998001409a8c2349d0637b8358, State: Initialized, Role: FOLLOWER
I20260928 16:36:53.095901 8577 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a578998001409a8c2349d0637b8358" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37581 } }
I20260928 16:36:53.096046 8577 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:36:53.096119 8577 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:36:53.096215 8577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:36:53.098115 8577 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a578998001409a8c2349d0637b8358" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37581 } }
I20260928 16:36:53.098452 8577 leader_election.cc:302] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9a578998001409a8c2349d0637b8358; no voters:
I20260928 16:36:53.098759 8577 leader_election.cc:288] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:36:53.098886 8581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:36:53.099128 8581 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [term 2 LEADER]: Becoming Leader. State: Replica: b9a578998001409a8c2349d0637b8358, State: Running, Role: LEADER
I20260928 16:36:53.099520 8581 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a578998001409a8c2349d0637b8358" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37581 } }
I20260928 16:36:53.099960 8577 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:36:53.100467 8581 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9a578998001409a8c2349d0637b8358. Latest consensus state: current_term: 2 leader_uuid: "b9a578998001409a8c2349d0637b8358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a578998001409a8c2349d0637b8358" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37581 } } }
I20260928 16:36:53.100653 8581 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [sys.catalog]: This master's current role is: LEADER
I20260928 16:36:53.100863 8583 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b9a578998001409a8c2349d0637b8358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9a578998001409a8c2349d0637b8358" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37581 } } }
I20260928 16:36:53.101079 8583 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358 [sys.catalog]: This master's current role is: LEADER
I20260928 16:36:53.101095 8587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:36:53.104463 8587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:36:53.105005 8577 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 16:36:53.105185 8587 catalog_manager.cc:1259] Loaded cluster ID: 56e5af54b54749f2a8242e30c87f6440
I20260928 16:36:53.105232 8587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:36:53.107611 8587 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:36:53.108268 8587 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358: Loaded TSK: 0
I20260928 16:36:53.111734 8587 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:36:53.191893 8579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:53.192176 8579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:53.192265 8579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:53.199285 8579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:53.199417 8579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:53.199548 8579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:36:53.207840 8579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.122.212:39261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:34627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=38131
--tserver_master_addrs=127.27.122.254:37581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:53.209210 8579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:53.210508 8579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:53.222963 8615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:53.223148 8614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:53.223399 8579 server_base.cc:1034] running on GCE node
W20260928 16:36:53.223878 8617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:53.224422 8579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:53.225227 8579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:53.226408 8579 hybrid_clock.cc:648] HybridClock initialized: now 1790613413226389 us; error 33 us; skew 500 ppm
Sep 28 16:36:53 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:53.230044 8579 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:36:53.231125 8579 webserver.cc:460] Webserver started at http://127.27.122.193:38131/ using document root <none> and password file <none>
I20260928 16:36:53.231832 8579 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:53.231966 8579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:36:53 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:53.257099 8579 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:36:53.259639 8625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:53.261005 8579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:36:53.261127 8579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "f2978040a6204764b4ae2d740b68b353"
format_stamp: "Formatted at 2026-09-28 16:36:52 on dist-test-slave-g9sh"
server_key: "c7e5eef41c46be91967d8d3e3d8559ba"
server_key_iv: "4febc18fa2977dc29475a768d4d2ed9b"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:53.261675 8579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:53.279390 8579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:53.282701 8579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:53.282907 8579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:53.283548 8579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:36:53.284461 8579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:36:53.284547 8579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:53.284626 8579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:36:53.284672 8579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:53.295629 8579 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:34627
I20260928 16:36:53.295806 8740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:34627 every 8 connection(s)
I20260928 16:36:53.296885 8579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:36:53.307098 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 8579
I20260928 16:36:53.307588 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.122.212:39261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:38683
--local_ip_for_outbound_sockets=127.27.122.194
--tserver_master_addrs=127.27.122.254:37581
--webserver_port=41575
--webserver_interface=127.27.122.194
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.122.212:39261/kms with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:36:53 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:36:53.320984 8744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:53.299681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37581 (local address 127.27.122.193:48557)
0928 16:36:53.301419 (+ 1738us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:53.301476 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:53.302586 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:53.306172 (+ 3586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:53.306181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:53.306557 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:53.307207 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:53.307227 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.309195 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.309199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:53.310582 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:53.310594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.311842 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.311846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:53.312050 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:53.312726 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:53.312747 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:53.314526 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:53.317301 (+ 2775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:53.317307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:53.317322 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:53.317672 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:53.318109 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:53.318113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:53.318115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:53.318282 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:53.319063 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:53.319069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:53.319382 (+ 313us) client_negotiation.cc:769] Sending connection context
0928 16:36:53.319591 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 16:36:53.319918 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1495,"thread_start_us":126,"threads_started":1}
I20260928 16:36:53.322257 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:53.299545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:48557 (local address 127.27.122.254:37581)
0928 16:36:53.301074 (+ 1529us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:53.301099 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:53.301621 (+ 522us) server_negotiation.cc:407] Connection header received
0928 16:36:53.304959 (+ 3338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:53.304991 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:53.305416 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:53.305766 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:53.307414 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.309037 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.310733 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.311471 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.311629 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:53.314743 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:53.314749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:53.314764 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:53.314796 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:53.317152 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:53.317819 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:53.317825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:53.317831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:53.317973 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:53.318391 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:53.318395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:53.318396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:53.318857 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:53.318974 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:53.321114 (+ 2140us) server_negotiation.cc:299] Negotiation successful
0928 16:36:53.321408 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1357,"thread_start_us":128,"threads_started":1}
I20260928 16:36:53.324781 8741 heartbeater.cc:347] Connected to a master server at 127.27.122.254:37581
I20260928 16:36:53.325062 8741 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:53.325663 8741 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
W20260928 16:36:53.328125 8741 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:37581 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:36:53.328251 8741 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:53.328438 8741 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
I20260928 16:36:53.329473 8741 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:53.329622 8741 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
W20260928 16:36:53.330379 8741 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:36:53.469638 8745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:53.469991 8745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:53.470055 8745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:53.476300 8745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:53.476377 8745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:53.476472 8745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:36:53.483990 8745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.122.212:39261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:38683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=41575
--tserver_master_addrs=127.27.122.254:37581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:53.485410 8745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:53.486740 8745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:53.498229 8753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:53.498423 8745 server_base.cc:1034] running on GCE node
W20260928 16:36:53.498202 8750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:53.498186 8751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:53.499177 8745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:53.499825 8745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:53.501019 8745 hybrid_clock.cc:648] HybridClock initialized: now 1790613413500995 us; error 40 us; skew 500 ppm
Sep 28 16:36:53 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:53.503998 8745 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:36:53.504833 8745 webserver.cc:460] Webserver started at http://127.27.122.194:41575/ using document root <none> and password file <none>
I20260928 16:36:53.505414 8745 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:53.505515 8745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:36:53 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:53.530411 8745 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260928 16:36:53.534438 8761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:53.536173 8745 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 16:36:53.536315 8745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "e30e45d0347f4146a6bf5bad220282bd"
format_stamp: "Formatted at 2026-09-28 16:36:52 on dist-test-slave-g9sh"
server_key: "da23bfafc1a692f24d7f6c05d6881308"
server_key_iv: "2cc16fb08672feb87588174b50d08d03"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:53.537099 8745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:53.572360 8745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:53.576112 8745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:53.576339 8745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:53.577106 8745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:36:53.578224 8745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:36:53.578291 8745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:53.578351 8745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:36:53.578380 8745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:53.593295 8745 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:38683
I20260928 16:36:53.595934 8745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:36:53.597740 8876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:38683 every 8 connection(s)
I20260928 16:36:53.598853 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 8745
I20260928 16:36:53.599339 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.122.212:39261/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:42187
--local_ip_for_outbound_sockets=127.27.122.195
--tserver_master_addrs=127.27.122.254:37581
--webserver_port=38679
--webserver_interface=127.27.122.195
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:46109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.122.212:39261/kms with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:36:53 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:36:53.623863 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:53.600162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:59219 (local address 127.27.122.254:37581)
0928 16:36:53.600337 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:53.600340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:53.606213 (+ 5873us) server_negotiation.cc:407] Connection header received
0928 16:36:53.607526 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:53.607530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:53.607590 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:53.607643 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:53.609242 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.610067 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.611782 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.612149 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.612227 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:53.618895 (+ 6668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:53.618899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:53.618902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:53.618931 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:53.620730 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:53.621367 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:53.621371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:53.621373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:53.621462 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:53.622098 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:53.622101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:53.622103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:53.622330 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:53.622433 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:53.623543 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0928 16:36:53.623695 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 16:36:53.625830 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:53.600673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37581 (local address 127.27.122.194:59219)
0928 16:36:53.602244 (+ 1571us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:53.606084 (+ 3840us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:53.607257 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:53.608037 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:53.608045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:53.608397 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:53.609056 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:53.609077 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.610256 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.610262 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:53.611637 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:53.611650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.612440 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.612444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:53.612639 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:53.617079 (+ 4440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:53.617101 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:53.618690 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:53.620866 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:53.620872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:53.620887 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:53.621243 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:53.621574 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:53.621577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:53.621579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:53.621766 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:53.622507 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:53.622513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:53.623229 (+ 716us) client_negotiation.cc:769] Sending connection context
0928 16:36:53.624337 (+ 1108us) client_negotiation.cc:240] Negotiation successful
0928 16:36:53.624699 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1346,"thread_start_us":160,"threads_started":1}
I20260928 16:36:53.627683 8877 heartbeater.cc:347] Connected to a master server at 127.27.122.254:37581
I20260928 16:36:53.627934 8877 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:53.628517 8877 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
W20260928 16:36:53.630343 8877 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:37581 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:36:53.630471 8877 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:53.630651 8877 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
I20260928 16:36:53.631803 8877 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:53.631970 8877 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
W20260928 16:36:53.632802 8877 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:36:53.736677 8879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:53.737047 8879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:53.737144 8879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:53.742488 8879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:53.742632 8879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:53.742791 8879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:36:53.750726 8879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:46109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.122.212:39261/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:42187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=38679
--tserver_master_addrs=127.27.122.254:37581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:53.752117 8879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:53.753518 8879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:53.765158 8889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:53.765239 8887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:53.765172 8886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:53.765671 8879 server_base.cc:1034] running on GCE node
I20260928 16:36:53.766281 8879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:53.766963 8879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:53.770221 8879 hybrid_clock.cc:648] HybridClock initialized: now 1790613413770186 us; error 41 us; skew 500 ppm
Sep 28 16:36:53 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:53.773211 8879 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:36:53.774209 8879 webserver.cc:460] Webserver started at http://127.27.122.195:38679/ using document root <none> and password file <none>
I20260928 16:36:53.774940 8879 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:53.775069 8879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 16:36:53 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:53.809412 8879 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.003s sys 0.000s
I20260928 16:36:53.812740 8897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:53.814354 8879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 16:36:53.814502 8879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "916c4cbac84249ffa0eba0962517f0b6"
format_stamp: "Formatted at 2026-09-28 16:36:52 on dist-test-slave-g9sh"
server_key: "c493fe3ead742afd8002a12a55b4155d"
server_key_iv: "141fee043358bbd7ebcccaba9d79a573"
server_key_version: "kuduclusterkey@0"
I20260928 16:36:53.815069 8879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:53.830451 8879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:53.834210 8879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:53.834439 8879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:53.835239 8879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:36:53.836335 8879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:36:53.836443 8879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:53.836558 8879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:36:53.836630 8879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:53.847955 8879 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:42187
I20260928 16:36:53.847971 9012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:42187 every 8 connection(s)
I20260928 16:36:53.849164 8879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:36:53.858163 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 8879
Sep 28 16:36:53 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613413, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:36:53.867772 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:53.851424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37581 (local address 127.27.122.195:56891)
0928 16:36:53.851874 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:53.851924 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:53.852813 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:53.853627 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:53.853639 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:53.853925 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:53.854515 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:53.854531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.855326 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.855329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:53.856306 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:53.856315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.857115 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.857118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:53.857314 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:53.857948 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:53.857966 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:53.860297 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:53.864428 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:53.864435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:53.864450 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:53.864803 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:53.865187 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:53.865191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:53.865194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:53.865379 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:53.865867 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:53.865873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:53.866193 (+ 320us) client_negotiation.cc:769] Sending connection context
0928 16:36:53.866402 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 16:36:53.866720 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":135,"threads_started":1}
I20260928 16:36:53.868446 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:53.850987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:56891 (local address 127.27.122.254:37581)
0928 16:36:53.851112 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:53.851117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:53.852021 (+ 904us) server_negotiation.cc:407] Connection header received
0928 16:36:53.853093 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:53.853097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:53.853156 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:53.853210 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:53.854661 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.855184 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.856473 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.856807 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.856888 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:53.860478 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:53.860482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:53.860485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:53.860513 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:53.864300 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:53.864946 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:53.864951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:53.864953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:53.865049 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:53.865495 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:53.865499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:53.865501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:53.865714 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:53.865800 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:53.868171 (+ 2371us) server_negotiation.cc:299] Negotiation successful
0928 16:36:53.868294 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:36:53.870114 9013 heartbeater.cc:347] Connected to a master server at 127.27.122.254:37581
I20260928 16:36:53.870374 9013 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:53.870963 9013 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
W20260928 16:36:53.872676 9013 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:37581 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:36:53.872799 9013 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:53.872980 9013 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
I20260928 16:36:53.873800 9013 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:53.873950 9013 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
W20260928 16:36:53.874692 9013 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:36:53.880409 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:53.858866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.27.122.254:37581)
0928 16:36:53.859160 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:53.859166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:53.859574 (+ 408us) server_negotiation.cc:407] Connection header received
0928 16:36:53.859683 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:53.859687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:53.859740 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:53.859850 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:53.863442 (+ 3592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.865118 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.866084 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:53.866414 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:53.866518 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:53.868271 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:53.868274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:53.868276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:53.868299 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:53.878275 (+ 9976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:53.878831 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:53.878835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:53.878837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:53.878935 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:53.879434 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:53.879438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:53.879440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:53.879664 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:53.879890 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:53.880087 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 16:36:53.880226 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":94,"threads_started":1}
Sep 28 16:36:54 dist-test-slave-g9sh krb5kdc[7523](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790613414, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:54.330615 8741 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:54.330945 8741 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
I20260928 16:36:54.332204 8540 ts_manager.cc:194] Registered new tserver with Master: f2978040a6204764b4ae2d740b68b353 (127.27.122.193:34627)
I20260928 16:36:54.334005 8540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:48557
Sep 28 16:36:54 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613414, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:54.633046 8877 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:54.633343 8877 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
I20260928 16:36:54.634161 8540 ts_manager.cc:194] Registered new tserver with Master: e30e45d0347f4146a6bf5bad220282bd (127.27.122.194:38683)
I20260928 16:36:54.634938 8540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:59219
I20260928 16:36:54.874912 9013 heartbeater.cc:464] Registering TS with master...
I20260928 16:36:54.875182 9013 heartbeater.cc:511] Master 127.27.122.254:37581 requested a full tablet report, sending...
I20260928 16:36:54.881122 8540 ts_manager.cc:194] Registered new tserver with Master: 916c4cbac84249ffa0eba0962517f0b6 (127.27.122.195:42187)
I20260928 16:36:54.881886 8540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:56891
Sep 28 16:36:54 dist-test-slave-g9sh krb5kdc[7523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613414, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for HTTP/127.27.122.212@KRBTEST.COM
I20260928 16:36:54.884258 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:36:54.907711 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:54.886520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.27.122.254:37581)
0928 16:36:54.892591 (+ 6071us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:54.892596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:54.892611 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:36:54.892650 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:54.892653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:54.892698 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:54.892784 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:54.893716 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.894529 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.896023 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.896366 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.896465 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:54.902455 (+ 5990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:54.902460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:54.902463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:54.902490 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:54.904246 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:54.904681 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:54.904685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:54.904687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:54.904787 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:54.905161 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:54.905164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:54.905166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:54.905407 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:54.905545 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:54.907397 (+ 1852us) server_negotiation.cc:299] Negotiation successful
0928 16:36:54.907538 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5988,"thread_start_us":98,"threads_started":1}
I20260928 16:36:54.911557 8540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:36:54.916539 8540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:36:54.941269 9042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:54.933350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:42187 (local address 127.0.0.1:59090)
0928 16:36:54.935943 (+ 2593us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:54.935956 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:54.936035 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:54.936714 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:54.936718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:54.936735 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:36:54.936836 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:54.936842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.938364 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.938368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:54.939515 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:54.939526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.940625 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.940630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:54.940749 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:54.940795 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 16:36:54.940884 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:36:54.941021 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2516,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260928 16:36:54.942886 9044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:54.933532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.27.122.195:42187)
0928 16:36:54.936268 (+ 2736us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:54.936276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:54.936300 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:36:54.936377 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:54.936382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:54.936569 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:54.937005 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:36:54.937179 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.938226 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.939664 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.940488 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.942450 (+ 1962us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:54.942598 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:54.942650 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 16:36:54.942744 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2622,"thread_start_us":105,"threads_started":1}
I20260928 16:36:54.943005 9046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:54.933930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47830 (local address 127.27.122.194:38683)
0928 16:36:54.935101 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:54.935107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:54.935141 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:36:54.935214 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:54.935219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:54.935410 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:54.935560 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:36:54.935987 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.937120 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.940400 (+ 3280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.941222 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.941316 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:54.941438 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:54.942219 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 16:36:54.942859 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1057,"thread_start_us":104,"threads_started":1}
I20260928 16:36:54.944958 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:54.932666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:38683 (local address 127.0.0.1:47830)
0928 16:36:54.933980 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:54.933994 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:54.934086 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:54.935566 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:54.935571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:54.935591 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:36:54.935850 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:54.935859 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.938147 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.938151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:54.940265 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:54.940277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.941962 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.941965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:54.942064 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:54.942086 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:36:54.944370 (+ 2284us) client_negotiation.cc:240] Negotiation successful
0928 16:36:54.944447 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1248,"thread_start_us":84,"threads_started":1}
I20260928 16:36:54.946470 8946 tablet_service.cc:1467] Processing CreateTablet for tablet 4f14d124f40a48ec9e90ff960f050e03 (DEFAULT_TABLE table=test-table [id=c096e1a7bada4538916dda05846761b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:36:54.949035 8810 tablet_service.cc:1467] Processing CreateTablet for tablet 4f14d124f40a48ec9e90ff960f050e03 (DEFAULT_TABLE table=test-table [id=c096e1a7bada4538916dda05846761b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:36:54.950134 8810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f14d124f40a48ec9e90ff960f050e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:36:54.953444 8946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f14d124f40a48ec9e90ff960f050e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:36:54.955823 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:54.932516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:34627 (local address 127.0.0.1:45652)
0928 16:36:54.932933 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:54.932986 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:54.933117 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:54.934807 (+ 1690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:54.934813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:54.934837 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:36:54.934984 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:54.934992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.937830 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.937834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:54.954078 (+ 16244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:54.954091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.955399 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.955403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:54.955509 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:54.955533 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:36:54.955606 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 16:36:54.955694 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":113,"threads_started":1}
I20260928 16:36:54.961324 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:54.933569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.27.122.193:34627)
0928 16:36:54.934404 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:54.934410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:54.934429 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:36:54.934493 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:54.934497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:54.934655 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:54.934818 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:36:54.935120 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.937697 (+ 2577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.954241 (+ 16544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:54.955072 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:54.955202 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:54.955328 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:54.961063 (+ 5735us) server_negotiation.cc:299] Negotiation successful
0928 16:36:54.961178 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":95,"threads_started":1}
I20260928 16:36:54.964830 8672 tablet_service.cc:1467] Processing CreateTablet for tablet 4f14d124f40a48ec9e90ff960f050e03 (DEFAULT_TABLE table=test-table [id=c096e1a7bada4538916dda05846761b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:36:54.967093 8672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f14d124f40a48ec9e90ff960f050e03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:36:54.970134 9047 tablet_bootstrap.cc:492] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6: Bootstrap starting.
I20260928 16:36:54.972492 9047 tablet_bootstrap.cc:654] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6: Neither blocks nor log segments found. Creating new log.
I20260928 16:36:54.973398 9047 log.cc:824] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:36:54.974095 9048 tablet_bootstrap.cc:492] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd: Bootstrap starting.
I20260928 16:36:54.976325 9048 tablet_bootstrap.cc:654] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd: Neither blocks nor log segments found. Creating new log.
I20260928 16:36:54.977233 9048 log.cc:824] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd: Log is configured to *not* fsync() on all Append() calls
I20260928 16:36:54.979780 9048 tablet_bootstrap.cc:492] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd: No bootstrap required, opened a new log
I20260928 16:36:54.979972 9048 ts_tablet_manager.cc:1397] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:36:54.984386 9048 raft_consensus.cc:348] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2978040a6204764b4ae2d740b68b353" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 34627 } } peers { permanent_uuid: "e30e45d0347f4146a6bf5bad220282bd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38683 } } peers { permanent_uuid: "916c4cbac84249ffa0eba0962517f0b6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42187 } }
I20260928 16:36:54.984776 9048 raft_consensus.cc:374] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:36:54.984929 9048 raft_consensus.cc:729] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e30e45d0347f4146a6bf5bad220282bd, State: Initialized, Role: FOLLOWER
I20260928 16:36:54.985523 9048 consensus_queue.cc:260] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2978040a6204764b4ae2d740b68b353" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 34627 } } peers { permanent_uuid: "e30e45d0347f4146a6bf5bad220282bd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38683 } } peers { permanent_uuid: "916c4cbac84249ffa0eba0962517f0b6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42187 } }
I20260928 16:36:54.986675 9048 ts_tablet_manager.cc:1428] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260928 16:36:54.987087 8877 heartbeater.cc:503] Master 127.27.122.254:37581 was elected leader, sending a full tablet report...
I20260928 16:36:54.990957 9051 tablet_bootstrap.cc:492] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353: Bootstrap starting.
I20260928 16:36:54.994102 9047 tablet_bootstrap.cc:492] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6: No bootstrap required, opened a new log
I20260928 16:36:54.994360 9047 ts_tablet_manager.cc:1397] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20260928 16:36:54.997931 9047 raft_consensus.cc:348] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2978040a6204764b4ae2d740b68b353" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 34627 } } peers { permanent_uuid: "e30e45d0347f4146a6bf5bad220282bd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38683 } } peers { permanent_uuid: "916c4cbac84249ffa0eba0962517f0b6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42187 } }
I20260928 16:36:54.998286 9047 raft_consensus.cc:374] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:36:54.998382 9047 raft_consensus.cc:729] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916c4cbac84249ffa0eba0962517f0b6, State: Initialized, Role: FOLLOWER
I20260928 16:36:54.998924 9047 consensus_queue.cc:260] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2978040a6204764b4ae2d740b68b353" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 34627 } } peers { permanent_uuid: "e30e45d0347f4146a6bf5bad220282bd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38683 } } peers { permanent_uuid: "916c4cbac84249ffa0eba0962517f0b6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42187 } }
I20260928 16:36:55.000077 9047 ts_tablet_manager.cc:1428] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:36:55.000464 9013 heartbeater.cc:503] Master 127.27.122.254:37581 was elected leader, sending a full tablet report...
I20260928 16:36:55.002406 9051 tablet_bootstrap.cc:654] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353: Neither blocks nor log segments found. Creating new log.
I20260928 16:36:55.003364 9051 log.cc:824] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353: Log is configured to *not* fsync() on all Append() calls
I20260928 16:36:55.008808 9051 tablet_bootstrap.cc:492] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353: No bootstrap required, opened a new log
I20260928 16:36:55.010027 9051 ts_tablet_manager.cc:1397] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260928 16:36:55.017938 9051 raft_consensus.cc:348] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2978040a6204764b4ae2d740b68b353" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 34627 } } peers { permanent_uuid: "e30e45d0347f4146a6bf5bad220282bd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38683 } } peers { permanent_uuid: "916c4cbac84249ffa0eba0962517f0b6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42187 } }
I20260928 16:36:55.018303 9051 raft_consensus.cc:374] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:36:55.018394 9051 raft_consensus.cc:729] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2978040a6204764b4ae2d740b68b353, State: Initialized, Role: FOLLOWER
I20260928 16:36:55.020033 9051 consensus_queue.cc:260] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2978040a6204764b4ae2d740b68b353" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 34627 } } peers { permanent_uuid: "e30e45d0347f4146a6bf5bad220282bd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38683 } } peers { permanent_uuid: "916c4cbac84249ffa0eba0962517f0b6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42187 } }
I20260928 16:36:55.020756 8741 heartbeater.cc:503] Master 127.27.122.254:37581 was elected leader, sending a full tablet report...
I20260928 16:36:55.021111 9051 ts_tablet_manager.cc:1428] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
W20260928 16:36:55.050386 8742 tablet.cc:2369] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:36:55.101176 9014 tablet.cc:2369] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:36:55.103065 8878 tablet.cc:2369] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:36:55.368983 9054 raft_consensus.cc:482] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:36:55.369223 9054 raft_consensus.cc:504] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2978040a6204764b4ae2d740b68b353" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 34627 } } peers { permanent_uuid: "e30e45d0347f4146a6bf5bad220282bd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38683 } } peers { permanent_uuid: "916c4cbac84249ffa0eba0962517f0b6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42187 } }
I20260928 16:36:55.370446 9054 leader_election.cc:288] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2978040a6204764b4ae2d740b68b353 (127.27.122.193:34627), e30e45d0347f4146a6bf5bad220282bd (127.27.122.194:38683)
I20260928 16:36:55.375438 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:55.370825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:54799 (local address 127.27.122.193:34627)
0928 16:36:55.370960 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:55.370965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:55.371165 (+ 200us) server_negotiation.cc:407] Connection header received
0928 16:36:55.371429 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:55.371437 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:55.371509 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:55.371579 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:36:55.371908 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:55.372667 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:55.374405 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:55.374810 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:55.374946 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:55.375005 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:55.375207 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 16:36:55.375307 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:36:55.375519 9065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:55.370762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:38683 (local address 127.27.122.195:57609)
0928 16:36:55.371120 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:55.371133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:55.371234 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:55.371520 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:55.371524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:55.371547 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:36:55.371711 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:55.371716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:55.372814 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:55.372818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:55.374224 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:55.374236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:55.375092 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:55.375095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:55.375200 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:55.375221 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:36:55.375297 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 16:36:55.375358 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260928 16:36:55.375538 9046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:55.371162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:57609 (local address 127.27.122.194:38683)
0928 16:36:55.371274 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:55.371278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:55.371293 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:36:55.371350 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:55.371353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:55.371401 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:55.371497 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:36:55.371831 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:55.372681 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:55.374367 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:55.374991 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:55.375113 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:55.375160 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:55.375327 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 16:36:55.375403 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 16:36:55.375962 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:55.370651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:34627 (local address 127.27.122.195:54799)
0928 16:36:55.371056 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:55.371070 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:55.371264 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:55.371633 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:55.371636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:55.371655 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:36:55.371781 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:55.371789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:55.372814 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:55.372818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:55.374261 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:55.374272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:55.374939 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:55.374943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:55.375042 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:55.375076 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 16:36:55.375746 (+ 670us) client_negotiation.cc:240] Negotiation successful
0928 16:36:55.375818 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":153,"spinlock_wait_cycles":17792,"thread_start_us":132,"threads_started":1}
I20260928 16:36:55.376248 8822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f14d124f40a48ec9e90ff960f050e03" candidate_uuid: "916c4cbac84249ffa0eba0962517f0b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30e45d0347f4146a6bf5bad220282bd" is_pre_election: true
I20260928 16:36:55.376576 8822 raft_consensus.cc:2454] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916c4cbac84249ffa0eba0962517f0b6 in term 0.
I20260928 16:36:55.376590 8695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f14d124f40a48ec9e90ff960f050e03" candidate_uuid: "916c4cbac84249ffa0eba0962517f0b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2978040a6204764b4ae2d740b68b353" is_pre_election: true
I20260928 16:36:55.376837 8695 raft_consensus.cc:2454] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916c4cbac84249ffa0eba0962517f0b6 in term 0.
I20260928 16:36:55.377130 8900 leader_election.cc:302] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916c4cbac84249ffa0eba0962517f0b6, e30e45d0347f4146a6bf5bad220282bd; no voters:
I20260928 16:36:55.377408 9054 raft_consensus.cc:2781] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:36:55.377473 9054 raft_consensus.cc:482] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:36:55.377501 9054 raft_consensus.cc:3037] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:36:55.378479 9054 raft_consensus.cc:504] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2978040a6204764b4ae2d740b68b353" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 34627 } } peers { permanent_uuid: "e30e45d0347f4146a6bf5bad220282bd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38683 } } peers { permanent_uuid: "916c4cbac84249ffa0eba0962517f0b6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42187 } }
I20260928 16:36:55.378863 9054 leader_election.cc:288] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [CANDIDATE]: Term 1 election: Requested vote from peers f2978040a6204764b4ae2d740b68b353 (127.27.122.193:34627), e30e45d0347f4146a6bf5bad220282bd (127.27.122.194:38683)
I20260928 16:36:55.379175 8695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f14d124f40a48ec9e90ff960f050e03" candidate_uuid: "916c4cbac84249ffa0eba0962517f0b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2978040a6204764b4ae2d740b68b353"
I20260928 16:36:55.379263 8822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f14d124f40a48ec9e90ff960f050e03" candidate_uuid: "916c4cbac84249ffa0eba0962517f0b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e30e45d0347f4146a6bf5bad220282bd"
I20260928 16:36:55.379302 8695 raft_consensus.cc:3037] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:36:55.379390 8822 raft_consensus.cc:3037] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:36:55.380520 8695 raft_consensus.cc:2454] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916c4cbac84249ffa0eba0962517f0b6 in term 1.
I20260928 16:36:55.380528 8822 raft_consensus.cc:2454] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916c4cbac84249ffa0eba0962517f0b6 in term 1.
I20260928 16:36:55.380792 8900 leader_election.cc:302] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916c4cbac84249ffa0eba0962517f0b6, e30e45d0347f4146a6bf5bad220282bd; no voters:
I20260928 16:36:55.381011 9054 raft_consensus.cc:2781] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:36:55.381261 9054 raft_consensus.cc:686] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [term 1 LEADER]: Becoming Leader. State: Replica: 916c4cbac84249ffa0eba0962517f0b6, State: Running, Role: LEADER
I20260928 16:36:55.381603 9054 consensus_queue.cc:237] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2978040a6204764b4ae2d740b68b353" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 34627 } } peers { permanent_uuid: "e30e45d0347f4146a6bf5bad220282bd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38683 } } peers { permanent_uuid: "916c4cbac84249ffa0eba0962517f0b6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42187 } }
I20260928 16:36:55.385177 8538 catalog_manager.cc:5495] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 916c4cbac84249ffa0eba0962517f0b6 (127.27.122.195). New cstate: current_term: 1 leader_uuid: "916c4cbac84249ffa0eba0962517f0b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2978040a6204764b4ae2d740b68b353" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 34627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e30e45d0347f4146a6bf5bad220282bd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "916c4cbac84249ffa0eba0962517f0b6" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42187 } health_report { overall_health: HEALTHY } } }
I20260928 16:36:55.504526 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:55.500323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.27.122.195:42187)
0928 16:36:55.500558 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:55.500572 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:55.500590 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:36:55.500704 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:55.500706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:55.500763 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:55.500962 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:36:55.501256 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:55.501907 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:55.502851 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:55.503165 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:55.503273 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:55.503654 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:36:55.504074 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:36:55.504167 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:55.504309 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 16:36:55.504360 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":66,"threads_started":1}
I20260928 16:36:55.510272 8822 raft_consensus.cc:1261] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd [term 1 FOLLOWER]: Refusing update from remote peer 916c4cbac84249ffa0eba0962517f0b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:36:55.510272 8695 raft_consensus.cc:1261] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353 [term 1 FOLLOWER]: Refusing update from remote peer 916c4cbac84249ffa0eba0962517f0b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:36:55.511039 9066 consensus_queue.cc:1035] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e30e45d0347f4146a6bf5bad220282bd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 38683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:36:55.511235 9054 consensus_queue.cc:1035] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2978040a6204764b4ae2d740b68b353" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 34627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:36:55.521422 9074 mvcc.cc:205] Tried to move back new op lower bound from 7334352549922299904 to 7334352549405720576. Current Snapshot: MvccSnapshot[applied={T|T < 7334352549922299904}]
I20260928 16:36:55.521804 9075 mvcc.cc:205] Tried to move back new op lower bound from 7334352549922299904 to 7334352549405720576. Current Snapshot: MvccSnapshot[applied={T|T < 7334352549922299904}]
I20260928 16:36:55.529814 8540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:36:55.529994 8540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:36:55.532570 8540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9a578998001409a8c2349d0637b8358: Sending DeleteTablet for 3 replicas of tablet 4f14d124f40a48ec9e90ff960f050e03
I20260928 16:36:55.533311 8672 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f14d124f40a48ec9e90ff960f050e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:36:55 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:45652
I20260928 16:36:55.533520 8810 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f14d124f40a48ec9e90ff960f050e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:36:55 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:47830
I20260928 16:36:55.533690 8946 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f14d124f40a48ec9e90ff960f050e03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:36:55 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:59090
I20260928 16:36:55.533802 9083 tablet_replica.cc:330] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353: stopping tablet replica
I20260928 16:36:55.534118 9084 tablet_replica.cc:330] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd: stopping tablet replica
I20260928 16:36:55.534096 9083 raft_consensus.cc:2229] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:36:55.534409 9083 raft_consensus.cc:2258] T 4f14d124f40a48ec9e90ff960f050e03 P f2978040a6204764b4ae2d740b68b353 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:36:55.534462 9084 raft_consensus.cc:2229] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:36:55.534711 9085 tablet_replica.cc:330] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6: stopping tablet replica
I20260928 16:36:55.534780 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 8579
I20260928 16:36:55.534793 9084 raft_consensus.cc:2258] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:36:55.535017 9085 raft_consensus.cc:2229] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:36:55.535357 9085 raft_consensus.cc:2258] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:36:55.536165 9084 ts_tablet_manager.cc:1905] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:36:55.537232 9085 ts_tablet_manager.cc:1905] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:36:55.539837 9084 ts_tablet_manager.cc:1918] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:36:55.539937 9084 log.cc:1196] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/4f14d124f40a48ec9e90ff960f050e03
I20260928 16:36:55.540198 9085 ts_tablet_manager.cc:1918] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:36:55.540278 9085 log.cc:1196] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/4f14d124f40a48ec9e90ff960f050e03
I20260928 16:36:55.540318 9084 ts_tablet_manager.cc:1939] T 4f14d124f40a48ec9e90ff960f050e03 P e30e45d0347f4146a6bf5bad220282bd: Deleting consensus metadata
I20260928 16:36:55.540572 9085 ts_tablet_manager.cc:1939] T 4f14d124f40a48ec9e90ff960f050e03 P 916c4cbac84249ffa0eba0962517f0b6: Deleting consensus metadata
I20260928 16:36:55.541595 8525 catalog_manager.cc:4841] TS e30e45d0347f4146a6bf5bad220282bd (127.27.122.194:38683): tablet 4f14d124f40a48ec9e90ff960f050e03 (table test-table [id=c096e1a7bada4538916dda05846761b5]) successfully deleted
I20260928 16:36:55.541792 8525 catalog_manager.cc:4841] TS 916c4cbac84249ffa0eba0962517f0b6 (127.27.122.195:42187): tablet 4f14d124f40a48ec9e90ff960f050e03 (table test-table [id=c096e1a7bada4538916dda05846761b5]) successfully deleted
W20260928 16:36:55.546711 8524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:34627 (error 108)
I20260928 16:36:55.547542 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 8745
I20260928 16:36:55.548802 9087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:55.547399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:34627 (local address 127.0.0.1:45656)
0928 16:36:55.548533 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:55.548622 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:34627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":68,"threads_started":1}
W20260928 16:36:55.549299 8524 catalog_manager.cc:4584] TS f2978040a6204764b4ae2d740b68b353 (127.27.122.193:34627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f14d124f40a48ec9e90ff960f050e03: Network error: Client connection negotiation failed: client connection to 127.27.122.193:34627: connect: Connection refused (error 111)
I20260928 16:36:55.555159 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 8879
I20260928 16:36:55.560675 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 8506
I20260928 16:36:55.567188 28139 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 16:36:55.902400 28139 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 16:36:55.902509 28139 mini_ranger.cc:66] Stopping Ranger...
I20260928 16:36:56.247488 28139 mini_ranger.cc:68] Stopped Ranger
2026-09-28 16:36:56.247 UTC [7555] LOG: received smart shutdown request
2026-09-28 16:36:56.251 UTC [7555] LOG: background worker "logical replication launcher" (PID 7566) exited with exit code 1
2026-09-28 16:36:56.251 UTC [7560] LOG: shutting down
2026-09-28 16:36:56.403 UTC [7555] LOG: database system is shut down
2026-09-28T16:36:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47608 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:36:56 dist-test-slave-g9sh krb5kdc[9096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:36:56 dist-test-slave-g9sh krb5kdc[9096](info): set up 2 sockets
Sep 28 16:36:56 dist-test-slave-g9sh krb5kdc[9096](info): commencing operation
krb5kdc: starting...
W20260928 16:36:57.563549 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:36:57 dist-test-slave-g9sh krb5kdc[9096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:36:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:36:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:36:57.681896 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:37475
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:37475,127.27.122.253:34707,127.27.122.252:39895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:36:57.792646 9109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:57.792939 9109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:57.793004 9109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:57.797084 9109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:36:57.797135 9109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:57.797158 9109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:57.797196 9109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:36:57.797227 9109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:36:57.802471 9109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:37475,127.27.122.253:34707,127.27.122.252:39895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:37475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:57.803614 9109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:57.804675 9109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:57.813462 9116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:57.813450 9118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:57.813548 9109 server_base.cc:1034] running on GCE node
W20260928 16:36:57.813464 9115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:57.814282 9109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:57.815333 9109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:57.816520 9109 hybrid_clock.cc:648] HybridClock initialized: now 1790613417816504 us; error 34 us; skew 500 ppm
Sep 28 16:36:57 dist-test-slave-g9sh krb5kdc[9096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:57.819268 9109 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:36:57.820163 9109 webserver.cc:460] Webserver started at http://127.27.122.254:37397/ using document root <none> and password file <none>
I20260928 16:36:57.820714 9109 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:57.820806 9109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:36:57.821090 9109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:36:57.823150 9109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "49ad92b35c5a4168a993f7dcc2fb3529"
format_stamp: "Formatted at 2026-09-28 16:36:57 on dist-test-slave-g9sh"
server_key: "643b4ea15defbf8a089df857dc5ab3f7"
server_key_iv: "ec7fb32dab9a8e8b10d14ed6fd793518"
server_key_version: "encryptionkey@0"
I20260928 16:36:57.823728 9109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "49ad92b35c5a4168a993f7dcc2fb3529"
format_stamp: "Formatted at 2026-09-28 16:36:57 on dist-test-slave-g9sh"
server_key: "643b4ea15defbf8a089df857dc5ab3f7"
server_key_iv: "ec7fb32dab9a8e8b10d14ed6fd793518"
server_key_version: "encryptionkey@0"
I20260928 16:36:57.827394 9109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 16:36:57.829864 9125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:57.831017 9109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:36:57.831162 9109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "49ad92b35c5a4168a993f7dcc2fb3529"
format_stamp: "Formatted at 2026-09-28 16:36:57 on dist-test-slave-g9sh"
server_key: "643b4ea15defbf8a089df857dc5ab3f7"
server_key_iv: "ec7fb32dab9a8e8b10d14ed6fd793518"
server_key_version: "encryptionkey@0"
I20260928 16:36:57.831274 9109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:57.845767 9109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:57.848951 9109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:57.849151 9109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:57.855952 9109 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:37475
I20260928 16:36:57.856226 9179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:37475 every 8 connection(s)
I20260928 16:36:57.856945 9109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:36:57.857519 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 9109
I20260928 16:36:57.857694 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:36:57.857988 28139 external_mini_cluster.cc:1428] Setting key 4e11648b77c595a022b7d27df67099dd
I20260928 16:36:57.859814 9180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:36:57.864354 9180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.122.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.253@KRBTEST.COM" created.
Sep 28 16:36:57 dist-test-slave-g9sh krb5kdc[9096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:36:57.879745 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:57.866598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.27.122.254:37475)
0928 16:36:57.867053 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:57.867072 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:57.867257 (+ 185us) server_negotiation.cc:407] Connection header received
0928 16:36:57.868340 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:57.868349 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:57.868573 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:57.868750 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:57.869652 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:57.870764 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:57.871926 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:57.872297 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:57.872473 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:57.874776 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:57.874787 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:57.874797 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:57.874818 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:57.876434 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:57.876954 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:57.876960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:57.876966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:57.877120 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:57.877441 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:57.877446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:57.877448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:57.877913 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:57.878053 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:57.878427 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 16:36:57.878675 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":316,"spinlock_wait_cycles":37120,"thread_start_us":164,"threads_started":1}
I20260928 16:36:57.879746 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:57.866598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37475 (local address 127.0.0.1:46356)
0928 16:36:57.867100 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:57.867149 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:57.868037 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:57.868778 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:57.868782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:57.869173 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:57.869494 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:57.869507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:57.870892 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:57.870899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:57.871804 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:57.871815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:57.872451 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:57.872455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:57.872569 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:57.873215 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:57.873232 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:57.874621 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:57.876578 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:57.876586 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:57.876597 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:57.876823 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:57.877224 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:57.877229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:57.877231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:57.877347 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:57.878024 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:57.878031 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:57.878194 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 16:36:57.878366 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 16:36:57.878755 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":160,"thread_start_us":137,"threads_started":1}
I20260928 16:36:57.882144 9180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:36:57.882917 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:57.882562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34707 (local address 127.0.0.1:52366)
0928 16:36:57.882771 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:57.882852 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260928 16:36:57.883128 9126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34707: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.253.keytab.
Entry for principal kudu/127.27.122.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.253.keytab.
I20260928 16:36:57.883996 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:57.883755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34707 (local address 127.0.0.1:52370)
0928 16:36:57.883863 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:57.883915 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 16:36:57.884181 9180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.253:34707: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34707: connect: Connection refused (error 111)
I20260928 16:36:57.884814 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.253:34707
--webserver_interface=127.27.122.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.253.keytab
--principal=kudu/127.27.122.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:37475,127.27.122.253:34707,127.27.122.252:39895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:36:57.933458 9180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } attempt: 1
I20260928 16:36:57.934391 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:57.934098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34707 (local address 127.0.0.1:52372)
0928 16:36:57.934249 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:57.934314 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 16:36:57.935130 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:57.934928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34707 (local address 127.0.0.1:52376)
0928 16:36:57.935018 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:57.935074 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 16:36:57.935254 9180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.253:34707: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34707: connect: Connection refused (error 111)
W20260928 16:36:57.994300 9187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:57.994609 9187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:57.994699 9187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:57.998735 9187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:36:57.998816 9187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:57.998852 9187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:57.998908 9187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:36:57.998944 9187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:36:58.003435 9180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } attempt: 2
I20260928 16:36:58.004280 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.003969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34707 (local address 127.0.0.1:52384)
0928 16:36:58.004131 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.004205 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 16:36:58.004420 9187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:37475,127.27.122.253:34707,127.27.122.252:39895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.253.keytab
--principal=kudu/127.27.122.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.253:34707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.122.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:58.005086 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.004820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34707 (local address 127.0.0.1:52390)
0928 16:36:58.004965 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.005018 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:36:58.005218 9180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.253:34707: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34707: connect: Connection refused (error 111)
I20260928 16:36:58.005684 9187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:58.006685 9187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:36:58.014981 9192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:58.014989 9195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:58.015152 9187 server_base.cc:1034] running on GCE node
W20260928 16:36:58.015329 9193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:58.015887 9187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:58.016987 9187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:58.018167 9187 hybrid_clock.cc:648] HybridClock initialized: now 1790613418018152 us; error 26 us; skew 500 ppm
Sep 28 16:36:58 dist-test-slave-g9sh krb5kdc[9096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:58.020874 9187 init.cc:377] Logged in from keytab as kudu/127.27.122.253@KRBTEST.COM (short username kudu)
I20260928 16:36:58.021756 9187 webserver.cc:460] Webserver started at http://127.27.122.253:38633/ using document root <none> and password file <none>
I20260928 16:36:58.022300 9187 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:58.022387 9187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:36:58.022616 9187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:36:58.024593 9187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/data/instance:
uuid: "27da3bb8478744628b92b794b09cca34"
format_stamp: "Formatted at 2026-09-28 16:36:58 on dist-test-slave-g9sh"
server_key: "badb65f5980f9c405d24dd1865c64ff9"
server_key_iv: "dbcadfa9d1158d3a49c2da931cd1239a"
server_key_version: "encryptionkey@0"
I20260928 16:36:58.025202 9187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/wal/instance:
uuid: "27da3bb8478744628b92b794b09cca34"
format_stamp: "Formatted at 2026-09-28 16:36:58 on dist-test-slave-g9sh"
server_key: "badb65f5980f9c405d24dd1865c64ff9"
server_key_iv: "dbcadfa9d1158d3a49c2da931cd1239a"
server_key_version: "encryptionkey@0"
I20260928 16:36:58.028844 9187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:36:58.031296 9202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:58.032413 9187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:36:58.032548 9187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/wal
uuid: "27da3bb8478744628b92b794b09cca34"
format_stamp: "Formatted at 2026-09-28 16:36:58 on dist-test-slave-g9sh"
server_key: "badb65f5980f9c405d24dd1865c64ff9"
server_key_iv: "dbcadfa9d1158d3a49c2da931cd1239a"
server_key_version: "encryptionkey@0"
I20260928 16:36:58.032666 9187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:58.050096 9187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:58.052848 9187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:58.053066 9187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:58.060483 9187 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.253:34707
I20260928 16:36:58.060775 9256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.253:34707 every 8 connection(s)
I20260928 16:36:58.061575 9187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/data/info.pb
I20260928 16:36:58.064289 9257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:36:58.068027 9257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:36:58.071134 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 9187
I20260928 16:36:58.071287 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-1/wal/instance
I20260928 16:36:58.071589 28139 external_mini_cluster.cc:1428] Setting key 90f14fdfb225b66a770ef7324fec65d3
Sep 28 16:36:58 dist-test-slave-g9sh krb5kdc[9096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.253@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.252@KRBTEST.COM" created.
I20260928 16:36:58.082913 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.069924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.27.122.254:37475)
0928 16:36:58.070083 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.070087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.070844 (+ 757us) server_negotiation.cc:407] Connection header received
0928 16:36:58.072022 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.072025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.072074 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.072155 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.073757 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.074619 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.075828 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.076071 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.076199 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.078652 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.078656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.078658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.078684 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.080685 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.081292 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.081296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.081298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.081383 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.081740 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.081744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.081745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.081931 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.082040 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.082610 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.082744 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:36:58.083722 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.070263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37475 (local address 127.0.0.1:46368)
0928 16:36:58.070692 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.070734 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:58.071790 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:58.072366 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:58.072378 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:58.072870 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:58.073582 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.073598 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.074786 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.074789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.075662 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.075672 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.076241 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.076245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.076427 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.077077 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:58.077093 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:58.078470 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:58.080809 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.080815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.080828 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.081185 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.081480 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.081484 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.081486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.081633 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.082039 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:58.082046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:58.082297 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 16:36:58.082485 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 16:36:58.082848 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260928 16:36:58.085270 9257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:36:58 dist-test-slave-g9sh krb5kdc[9096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.253@KRBTEST.COM for kudu/127.27.122.253@KRBTEST.COM
I20260928 16:36:58.096441 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.085795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34707 (local address 127.0.0.1:52398)
0928 16:36:58.085886 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.085898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:58.085959 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:58.086584 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:58.086587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:58.086762 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:58.086833 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.086839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.087808 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.087811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.088329 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.088337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.088961 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.088965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.089058 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.089524 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:58.089536 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:58.092580 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:58.094787 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.094791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.094793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.094999 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.095307 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.095310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.095311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.095414 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.095881 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:58.095884 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:58.096003 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 16:36:58.096109 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 16:36:58.096240 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 16:36:58.096410 9180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } attempt: 3
I20260928 16:36:58.096467 9263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.085945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.27.122.253:34707)
0928 16:36:58.086263 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.086271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.086289 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:36:58.086321 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.086325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.086462 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.086607 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.086944 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.087681 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.088444 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.088821 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.088961 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.092735 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.092743 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.092754 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.092774 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.094667 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.095108 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.095114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.095116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.095208 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.095510 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.095513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.095514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.095757 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.095898 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.096213 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.096333 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":144,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.252.keytab.
Entry for principal kudu/127.27.122.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.252.keytab.
I20260928 16:36:58.098016 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.252:39895
--webserver_interface=127.27.122.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.252.keytab
--principal=kudu/127.27.122.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:41201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:37475,127.27.122.253:34707,127.27.122.252:39895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:36:58.098805 9257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:36:58.099637 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.099285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56582)
0928 16:36:58.099460 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.099541 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260928 16:36:58.100318 9204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
I20260928 16:36:58.101377 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.101042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56598)
0928 16:36:58.101226 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.101281 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":31}
W20260928 16:36:58.101519 9257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:39895: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
Sep 28 16:36:58 dist-test-slave-g9sh krb5kdc[9096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for kudu/127.27.122.253@KRBTEST.COM
I20260928 16:36:58.106317 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.096906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34707 (local address 127.0.0.1:52400)
0928 16:36:58.097130 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.097148 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:58.097232 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:58.098488 (+ 1256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:58.098490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:58.098637 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:58.098761 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.098768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.099834 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.099838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.100449 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.100461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.101047 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.101051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.101174 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.101633 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:58.101662 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:58.102896 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:58.104710 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.104714 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.104716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.104969 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.105268 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.105272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.105273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.105363 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.105803 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:58.105807 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:58.105907 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 16:36:58.106014 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 16:36:58.106158 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 16:36:58.106331 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.097007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.27.122.253:34707)
0928 16:36:58.097473 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.097481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.097491 (+ 10us) server_negotiation.cc:407] Connection header received
0928 16:36:58.097519 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.097521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.097552 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.097660 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.098896 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.099666 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.100566 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.100915 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.101034 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.103040 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.103044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.103046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.103077 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.104586 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.105077 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.105081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.105083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.105171 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.105470 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.105475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.105477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.105693 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.105801 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.106066 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.106180 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":112,"threads_started":1}
I20260928 16:36:58.107117 9180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:36:58.107813 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.107576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56612)
0928 16:36:58.107689 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.107739 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 16:36:58.108511 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.108267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56616)
0928 16:36:58.108406 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.108458 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260928 16:36:58.108656 9180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:39895: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
I20260928 16:36:58.139828 9180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } attempt: 1
I20260928 16:36:58.140715 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.140394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56624)
0928 16:36:58.140548 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.140626 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 16:36:58.141618 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.141389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56632)
0928 16:36:58.141505 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.141558 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 16:36:58.141806 9180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:39895: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
I20260928 16:36:58.150796 9257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } attempt: 1
I20260928 16:36:58.151749 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.151367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56644)
0928 16:36:58.151544 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.151648 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 16:36:58.152887 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.152604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56656)
0928 16:36:58.152756 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.152815 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 16:36:58.153118 9257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:39895: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
W20260928 16:36:58.211864 9265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:36:58.212167 9265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:36:58.212244 9265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:36:58.216287 9265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:36:58.216382 9265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:36:58.216418 9265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:36:58.216459 9265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:36:58.216490 9265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:36:58.216984 9180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } attempt: 2
I20260928 16:36:58.217821 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.217517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56670)
0928 16:36:58.217664 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.217739 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 16:36:58.218554 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.218341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56686)
0928 16:36:58.218450 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.218500 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 16:36:58.218704 9180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:39895: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
I20260928 16:36:58.221269 9257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } attempt: 2
I20260928 16:36:58.221985 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.221690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56700)
0928 16:36:58.221829 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.221890 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 16:36:58.222571 9265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:41201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:37475,127.27.122.253:34707,127.27.122.252:39895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.252.keytab
--principal=kudu/127.27.122.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.252:39895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.122.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:36:58.222652 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.222443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56702)
0928 16:36:58.222543 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.222600 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 16:36:58.222841 9257 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:39895: Network error: Client connection negotiation failed: client connection to 127.27.122.252:39895: connect: Connection refused (error 111)
I20260928 16:36:58.223704 9265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:36:58.224718 9265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:36:58.233253 9265 server_base.cc:1034] running on GCE node
W20260928 16:36:58.233222 9274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:58.233170 9271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:36:58.233191 9272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:36:58.234067 9265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:36:58.235260 9265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:36:58.236441 9265 hybrid_clock.cc:648] HybridClock initialized: now 1790613418236423 us; error 33 us; skew 500 ppm
Sep 28 16:36:58 dist-test-slave-g9sh krb5kdc[9096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:36:58.239323 9265 init.cc:377] Logged in from keytab as kudu/127.27.122.252@KRBTEST.COM (short username kudu)
I20260928 16:36:58.240185 9265 webserver.cc:460] Webserver started at http://127.27.122.252:36029/ using document root <none> and password file <none>
I20260928 16:36:58.240751 9265 fs_manager.cc:359] Metadata directory not provided
I20260928 16:36:58.240841 9265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:36:58.241096 9265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:36:58.243158 9265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/data/instance:
uuid: "b59dfe474a794d24a13ea987911b9ca8"
format_stamp: "Formatted at 2026-09-28 16:36:58 on dist-test-slave-g9sh"
server_key: "d7841f7a71ed9f3a8d67de78aff49845"
server_key_iv: "e2e87eb9aba3360cc43ab3669a1e2dac"
server_key_version: "encryptionkey@0"
I20260928 16:36:58.244020 9265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/wal/instance:
uuid: "b59dfe474a794d24a13ea987911b9ca8"
format_stamp: "Formatted at 2026-09-28 16:36:58 on dist-test-slave-g9sh"
server_key: "d7841f7a71ed9f3a8d67de78aff49845"
server_key_iv: "e2e87eb9aba3360cc43ab3669a1e2dac"
server_key_version: "encryptionkey@0"
I20260928 16:36:58.247857 9265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 16:36:58.250340 9281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:36:58.251539 9265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:36:58.251664 9265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/wal
uuid: "b59dfe474a794d24a13ea987911b9ca8"
format_stamp: "Formatted at 2026-09-28 16:36:58 on dist-test-slave-g9sh"
server_key: "d7841f7a71ed9f3a8d67de78aff49845"
server_key_iv: "e2e87eb9aba3360cc43ab3669a1e2dac"
server_key_version: "encryptionkey@0"
I20260928 16:36:58.251770 9265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:36:58.273910 9265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:36:58.276664 9265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:36:58.276865 9265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:36:58.283507 9265 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.252:39895
I20260928 16:36:58.283785 9334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.252:39895 every 8 connection(s)
I20260928 16:36:58.284471 9265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/data/info.pb
I20260928 16:36:58.284914 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 9265
I20260928 16:36:58.285097 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790613255639823-28139-0/minicluster-data/master-2/wal/instance
I20260928 16:36:58.285380 28139 external_mini_cluster.cc:1428] Setting key fdae35505bc7b510a74df45285deb26f
I20260928 16:36:58.287479 9335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:36:58.291735 9335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:36:58 dist-test-slave-g9sh krb5kdc[9096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.252@KRBTEST.COM
Sep 28 16:36:58 dist-test-slave-g9sh krb5kdc[9096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.252@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:36:58.300263 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.286734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.27.122.252:39895)
0928 16:36:58.287135 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.287160 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.287195 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:36:58.287894 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.287914 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.288258 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.288536 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.289453 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.290492 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.291653 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.292236 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.292358 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.295183 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.295188 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.295196 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.295220 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.296852 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.297454 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.297463 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.297471 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.297614 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.297906 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.297910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.297912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.298384 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.298534 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.298894 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.299206 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":147,"threads_started":1}
I20260928 16:36:58.302661 9184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.293715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.27.122.254:37475)
0928 16:36:58.293878 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.293885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.294138 (+ 253us) server_negotiation.cc:407] Connection header received
0928 16:36:58.294246 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.294249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.294304 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.294400 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.295157 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.295635 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.296492 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.296725 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.296839 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.299244 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.299248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.299251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.299278 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.300765 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.301271 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.301274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.301275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.301347 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.301655 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.301665 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.301666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.301831 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.301936 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.302383 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.302491 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:36:58.302687 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.293608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37475 (local address 127.0.0.1:46374)
0928 16:36:58.293994 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.294041 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:58.294161 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:58.294448 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:58.294452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:58.294873 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:58.294999 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.295005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.295766 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.295769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.296364 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.296372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.296840 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.296844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.296999 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.297509 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:58.297529 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:58.299117 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:58.300864 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.300871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.300876 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.301179 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.301447 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.301450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.301452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.301541 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.301953 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:58.301961 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:58.302140 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 16:36:58.302311 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 16:36:58.302492 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":125,"threads_started":1}
I20260928 16:36:58.304056 9335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:36:58 dist-test-slave-g9sh krb5kdc[9096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
Sep 28 16:36:58 dist-test-slave-g9sh krb5kdc[9096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.252@KRBTEST.COM for kudu/127.27.122.253@KRBTEST.COM
I20260928 16:36:58.311966 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.302660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.27.122.254:37475)
0928 16:36:58.303027 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.303035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.303049 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:36:58.303167 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.303170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.303215 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.303305 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.304042 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.304794 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.305814 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.306097 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.306224 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.308608 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.308613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.308615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.308647 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.310363 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.310791 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.310794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.310796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.310876 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.311157 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.311162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.311164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.311333 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.311538 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.311678 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.311792 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":149,"threads_started":1}
I20260928 16:36:58.312428 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.304470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34707 (local address 127.0.0.1:52402)
0928 16:36:58.304606 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.304619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:58.304686 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:58.304975 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:58.304978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:58.305111 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:58.305201 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.305206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.306168 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.306173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.306838 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.306847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.307334 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.307337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.307433 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.307817 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:58.307838 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:58.309158 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:58.310858 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.310862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.310864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.311054 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.311354 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.311357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.311359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.311441 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.311942 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:58.311945 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:58.312046 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 16:36:58.312174 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 16:36:58.312293 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 16:36:58.312479 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.304585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.27.122.253:34707)
0928 16:36:58.304698 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.304704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.304732 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:36:58.304783 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.304785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.304826 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.304943 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.305315 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.306063 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.306957 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.307220 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.307334 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.309290 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.309294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.309297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.309321 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.310745 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.311168 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.311171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.311173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.311255 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.311531 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.311535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.311537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.311729 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.311838 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.312256 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.312372 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 16:36:58.313470 9335 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:36:58.314024 9257 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } attempt: 3
I20260928 16:36:58.317881 9180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } attempt: 3
Sep 28 16:36:58 dist-test-slave-g9sh krb5kdc[9096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.252@KRBTEST.COM for kudu/127.27.122.252@KRBTEST.COM
Sep 28 16:36:58 dist-test-slave-g9sh krb5kdc[9096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.253@KRBTEST.COM for kudu/127.27.122.252@KRBTEST.COM
Sep 28 16:36:58 dist-test-slave-g9sh krb5kdc[9096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.253@KRBTEST.COM
I20260928 16:36:58.323894 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.313904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56708)
0928 16:36:58.314011 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.314023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:58.314095 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:58.314343 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:58.314346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:58.314506 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:58.314590 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.314600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.315478 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.315481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.316299 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.316313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.316848 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.316851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.317015 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.317410 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:58.317421 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:58.319503 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:58.321448 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.321450 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.321452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.321680 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.321917 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.321922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.321924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.322008 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.323362 (+ 1354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:58.323364 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:58.323452 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 16:36:58.323552 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 16:36:58.323714 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":24}
Sep 28 16:36:58 dist-test-slave-g9sh krb5kdc[9096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613417, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for kudu/127.27.122.252@KRBTEST.COM
I20260928 16:36:58.324591 9338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.314053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56708 (local address 127.27.122.252:39895)
0928 16:36:58.314173 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.314177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.314187 (+ 10us) server_negotiation.cc:407] Connection header received
0928 16:36:58.314213 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.314216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.314252 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.314332 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.314694 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.315374 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.316410 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.316751 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.316851 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.319653 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.319657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.319660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.319682 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.321343 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.321767 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.321770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.321772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.321836 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.322116 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.322120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.322122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.323253 (+ 1131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.324141 (+ 888us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.324208 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.324444 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":35}
I20260928 16:36:58.325140 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.313126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.27.122.253:34707)
0928 16:36:58.313239 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.313245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.313412 (+ 167us) server_negotiation.cc:407] Connection header received
0928 16:36:58.313522 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.313525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.313563 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.313657 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.314418 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.315037 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.315948 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.316168 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.316269 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.321425 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.321428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.321431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.321454 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.323136 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.323612 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.323616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.323617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.323702 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.324016 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.324020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.324022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.324250 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.324428 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.324800 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.324962 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 16:36:58.328717 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.318260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56738)
0928 16:36:58.318421 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.318434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:58.318508 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:58.319293 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:58.319298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:58.319459 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:58.319535 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.319540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.320572 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.320577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.321446 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.321457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.322072 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.322076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.322181 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.322846 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:58.322865 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:58.324263 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:58.327419 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.327421 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.327423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.327608 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.327888 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.327890 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.327891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.327962 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.328346 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:58.328348 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:58.328421 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 16:36:58.328496 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 16:36:58.328591 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 16:36:58.328821 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.318712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.27.122.252:39895)
0928 16:36:58.319026 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.319036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.319051 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:36:58.319087 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.319089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.319129 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.319633 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.319736 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.320443 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.321579 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.321942 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.322358 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.324406 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.324409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.324412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.324435 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.327316 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.327714 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.327717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.327719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.327798 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.328054 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.328057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.328059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.328256 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.328348 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.328569 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.328675 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":126,"threads_started":1}
I20260928 16:36:58.328836 9335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8: Bootstrap starting.
I20260928 16:36:58.329259 9259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.314381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:39895 (local address 127.0.0.1:56724)
0928 16:36:58.314494 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 16:36:58.314508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:36:58.314573 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:36:58.315171 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:36:58.315174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:36:58.315339 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:36:58.315438 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.315445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.316805 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.316810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.317557 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:36:58.317567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.318137 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.318140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:36:58.318228 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.318682 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:36:58.318697 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:36:58.320696 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:36:58.324257 (+ 3561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.324261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.324263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.324591 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.324926 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:36:58.324929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:36:58.324931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:36:58.325019 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:36:58.325494 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:36:58.325497 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:36:58.325607 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 16:36:58.325718 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 16:36:58.325850 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 16:36:58.330708 9341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.314642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56724 (local address 127.27.122.252:39895)
0928 16:36:58.314961 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.314969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.314981 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:36:58.315011 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.315014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.315050 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.315133 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.315529 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.316714 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.317671 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.318018 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.318143 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.320837 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.320841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.320843 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.320869 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.323884 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.324720 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.324723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.324725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.324821 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.325181 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.325184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.325186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.325408 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.325496 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.330108 (+ 4612us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.330235 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
I20260928 16:36:58.331504 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:36:58.331945 9335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8: Neither blocks nor log segments found. Creating new log.
I20260928 16:36:58.332005 9180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529: Bootstrap starting.
I20260928 16:36:58.332823 9335 log.cc:824] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8: Log is configured to *not* fsync() on all Append() calls
I20260928 16:36:58.333460 9257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34: Bootstrap starting.
I20260928 16:36:58.334725 9180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529: Neither blocks nor log segments found. Creating new log.
I20260928 16:36:58.335639 9180 log.cc:824] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529: Log is configured to *not* fsync() on all Append() calls
I20260928 16:36:58.336045 9257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34: Neither blocks nor log segments found. Creating new log.
I20260928 16:36:58.337069 9257 log.cc:824] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34: Log is configured to *not* fsync() on all Append() calls
I20260928 16:36:58.337769 9180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529: No bootstrap required, opened a new log
I20260928 16:36:58.339185 9257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34: No bootstrap required, opened a new log
I20260928 16:36:58.342691 9180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } }
I20260928 16:36:58.342922 9180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:36:58.342984 9180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49ad92b35c5a4168a993f7dcc2fb3529, State: Initialized, Role: FOLLOWER
I20260928 16:36:58.343641 9180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } }
I20260928 16:36:58.343629 9257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } }
I20260928 16:36:58.343883 9257 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:36:58.343983 9257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27da3bb8478744628b92b794b09cca34, State: Initialized, Role: FOLLOWER
I20260928 16:36:58.344194 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.333916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.27.122.254:37475)
0928 16:36:58.334039 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.334043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.334360 (+ 317us) server_negotiation.cc:407] Connection header received
0928 16:36:58.334531 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.334534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.334583 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.334651 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.335690 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.336430 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.337558 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.337899 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.337974 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.339755 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.339759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.339761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.339785 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.342277 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.342785 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.342788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.342791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.342882 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.343213 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.343216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.343218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.343421 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.343533 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.343904 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.344031 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:36:58.344458 9257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } }
I20260928 16:36:58.344580 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.334447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.27.122.252:39895)
0928 16:36:58.334540 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.334544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.335793 (+ 1249us) server_negotiation.cc:407] Connection header received
0928 16:36:58.336030 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.336034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.336082 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.336147 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.337238 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.337907 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.338968 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.339312 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.339388 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.341016 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.341020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.341023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.341048 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.342660 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.343163 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.343167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.343168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.343256 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.343596 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.343599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.343601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.343802 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.343942 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.344328 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.344438 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 16:36:58.345307 9180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:36:58.346521 9257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:36:58.346971 9355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } } }
I20260928 16:36:58.347198 9355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:36:58.350381 9335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8: No bootstrap required, opened a new log
W20260928 16:36:58.351665 9367 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:36:58.352051 9367 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:36:58.351938 9354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } } }
I20260928 16:36:58.352291 9354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:36:58.350658 9109 master_runner.cc:186] Error getting master registration for 127.27.122.253:34707: OK, instance_id { permanent_uuid: "27da3bb8478744628b92b794b09cca34" instance_seqno: 1790613418058500 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:36:58.354138 9109 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:36:58.354601 9264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:36:58.335185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.27.122.253:34707)
0928 16:36:58.335280 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 16:36:58.335283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:36:58.345667 (+ 10384us) server_negotiation.cc:407] Connection header received
0928 16:36:58.345839 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:36:58.345842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:36:58.345893 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:36:58.345947 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:36:58.346828 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.347658 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.348697 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:36:58.349020 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:36:58.349118 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:36:58.350707 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:36:58.350711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:36:58.350714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:36:58.350739 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:36:58.352425 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.352945 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.352948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.352950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.353035 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:36:58.353340 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:36:58.353343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:36:58.353345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:36:58.353550 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:36:58.353676 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 16:36:58.354348 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 16:36:58.354456 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":37}
W20260928 16:36:58.355203 9131 tablet.cc:2369] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:36:58.354885 9335 raft_consensus.cc:348] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } }
I20260928 16:36:58.355855 9335 raft_consensus.cc:374] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:36:58.355917 9335 raft_consensus.cc:729] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59dfe474a794d24a13ea987911b9ca8, State: Initialized, Role: FOLLOWER
I20260928 16:36:58.356330 9335 consensus_queue.cc:260] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } }
I20260928 16:36:58.356745 9375 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } } }
I20260928 16:36:58.356865 9375 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:36:58.358091 9335 sys_catalog.cc:564] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:36:58.364632 9379 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 16:36:58.364729 9187 master_runner.cc:263] No leader master found from master 27da3bb8478744628b92b794b09cca34
W20260928 16:36:58.364753 9379 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:36:58.364820 9187 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:36:58.366680 9265 master_runner.cc:263] No leader master found from master b59dfe474a794d24a13ea987911b9ca8
I20260928 16:36:58.366747 9265 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 16:36:58.367349 9391 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:36:58.367431 9391 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:36:58.396989 9354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:36:58.397198 9354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } }
I20260928 16:36:58.398222 9354 leader_election.cc:288] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27da3bb8478744628b92b794b09cca34 (127.27.122.253:34707), b59dfe474a794d24a13ea987911b9ca8 (127.27.122.252:39895)
I20260928 16:36:58.398936 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b59dfe474a794d24a13ea987911b9ca8" is_pre_election: true
I20260928 16:36:58.399277 9308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49ad92b35c5a4168a993f7dcc2fb3529 in term 0.
I20260928 16:36:58.399281 9230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27da3bb8478744628b92b794b09cca34" is_pre_election: true
I20260928 16:36:58.399564 9230 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49ad92b35c5a4168a993f7dcc2fb3529 in term 0.
I20260928 16:36:58.400019 9127 leader_election.cc:302] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ad92b35c5a4168a993f7dcc2fb3529, b59dfe474a794d24a13ea987911b9ca8; no voters:
I20260928 16:36:58.400600 9354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:36:58.400694 9354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:36:58.400736 9354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:36:58.402096 9354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } }
I20260928 16:36:58.402415 9354 leader_election.cc:288] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [CANDIDATE]: Term 1 election: Requested vote from peers 27da3bb8478744628b92b794b09cca34 (127.27.122.253:34707), b59dfe474a794d24a13ea987911b9ca8 (127.27.122.252:39895)
I20260928 16:36:58.402993 9308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b59dfe474a794d24a13ea987911b9ca8"
I20260928 16:36:58.403111 9308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:36:58.404608 9308 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ad92b35c5a4168a993f7dcc2fb3529 in term 1.
I20260928 16:36:58.404983 9127 leader_election.cc:302] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49ad92b35c5a4168a993f7dcc2fb3529, b59dfe474a794d24a13ea987911b9ca8; no voters:
I20260928 16:36:58.405463 9230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27da3bb8478744628b92b794b09cca34"
I20260928 16:36:58.405584 9230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:36:58.406989 9230 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49ad92b35c5a4168a993f7dcc2fb3529 in term 1.
I20260928 16:36:58.409708 9354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:36:58.409986 9354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [term 1 LEADER]: Becoming Leader. State: Replica: 49ad92b35c5a4168a993f7dcc2fb3529, State: Running, Role: LEADER
I20260928 16:36:58.410372 9354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } }
I20260928 16:36:58.412189 9354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ad92b35c5a4168a993f7dcc2fb3529. Latest consensus state: current_term: 1 leader_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } } }
I20260928 16:36:58.412352 9354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [sys.catalog]: This master's current role is: LEADER
I20260928 16:36:58.414254 9395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:36:58.415246 9395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:36:58.423256 9308 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [term 1 FOLLOWER]: Refusing update from remote peer 49ad92b35c5a4168a993f7dcc2fb3529: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:36:58.423996 9393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:36:58.428668 9375 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ad92b35c5a4168a993f7dcc2fb3529. Latest consensus state: current_term: 1 leader_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } } }
I20260928 16:36:58.428805 9375 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:36:58.432205 9230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [term 1 FOLLOWER]: Refusing update from remote peer 49ad92b35c5a4168a993f7dcc2fb3529: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:36:58.432873 9393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:36:58.437619 9355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49ad92b35c5a4168a993f7dcc2fb3529. Latest consensus state: current_term: 1 leader_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } } }
I20260928 16:36:58.437739 9355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:36:58.441781 9354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } } }
I20260928 16:36:58.441896 9354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [sys.catalog]: This master's current role is: LEADER
I20260928 16:36:58.443128 9375 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } } }
I20260928 16:36:58.443243 9375 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:36:58.443602 9396 mvcc.cc:205] Tried to move back new op lower bound from 7334352561854332928 to 7334352561812185088. Current Snapshot: MvccSnapshot[applied={T|T < 7334352561854332928}]
I20260928 16:36:58.443910 9397 mvcc.cc:205] Tried to move back new op lower bound from 7334352561854332928 to 7334352561812185088. Current Snapshot: MvccSnapshot[applied={T|T < 7334352561854332928}]
I20260928 16:36:58.445887 9393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } } }
I20260928 16:36:58.446002 9393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529 [sys.catalog]: This master's current role is: LEADER
I20260928 16:36:58.447012 9355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49ad92b35c5a4168a993f7dcc2fb3529" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37475 } } peers { permanent_uuid: "27da3bb8478744628b92b794b09cca34" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34707 } } peers { permanent_uuid: "b59dfe474a794d24a13ea987911b9ca8" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 39895 } } }
I20260928 16:36:58.447113 9355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:36:58.447975 9399 mvcc.cc:205] Tried to move back new op lower bound from 7334352561854332928 to 7334352561812185088. Current Snapshot: MvccSnapshot[applied={T|T < 7334352561854332928}]
I20260928 16:36:58.451808 9395 catalog_manager.cc:1347] Generated new cluster ID: a4eb3503021f4e90b78360ded341a91a
I20260928 16:36:58.451862 9395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:36:58.490969 9395 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:36:58.492456 9395 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:36:58.511029 9395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49ad92b35c5a4168a993f7dcc2fb3529: Generated new TSK 0
I20260928 16:36:58.512281 9395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:36:59.366291 9379 catalog_manager.cc:1259] Loaded cluster ID: a4eb3503021f4e90b78360ded341a91a
I20260928 16:36:59.366406 9379 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34: loading cluster ID for follower catalog manager: success
I20260928 16:36:59.368211 9379 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34: acquiring CA information for follower catalog manager: success
I20260928 16:36:59.368561 9391 catalog_manager.cc:1259] Loaded cluster ID: a4eb3503021f4e90b78360ded341a91a
I20260928 16:36:59.368659 9391 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8: loading cluster ID for follower catalog manager: success
I20260928 16:36:59.369544 9379 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27da3bb8478744628b92b794b09cca34: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:36:59.370329 9391 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8: acquiring CA information for follower catalog manager: success
I20260928 16:36:59.371376 9391 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b59dfe474a794d24a13ea987911b9ca8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:36:59.590480 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 9109
I20260928 16:36:59.597503 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 9187
I20260928 16:36:59.603907 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 9265
2026-09-28T16:36:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3080 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:36:59 dist-test-slave-g9sh krb5kdc[9413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:36:59 dist-test-slave-g9sh krb5kdc[9413](info): set up 2 sockets
Sep 28 16:36:59 dist-test-slave-g9sh krb5kdc[9413](info): commencing operation
krb5kdc: starting...
W20260928 16:37:00.655571 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:37:00 dist-test-slave-g9sh krb5kdc[9413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:37:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:37:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:37:00.779268 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:45059
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:45059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:37:00.888566 9427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:00.888814 9427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:00.888854 9427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:00.892776 9427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:37:00.892827 9427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:00.892846 9427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:00.892863 9427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:37:00.892877 9427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:37:00.898043 9427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:45059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:45059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:00.898974 9427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:00.899943 9427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:37:00.908527 9427 server_base.cc:1034] running on GCE node
W20260928 16:37:00.908439 9435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:00.908391 9433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:00.908391 9432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:00.909404 9427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:00.910542 9427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:00.911738 9427 hybrid_clock.cc:648] HybridClock initialized: now 1790613420911671 us; error 83 us; skew 500 ppm
Sep 28 16:37:00 dist-test-slave-g9sh krb5kdc[9413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:00.920344 9427 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:37:00.921382 9427 webserver.cc:460] Webserver started at http://127.27.122.254:34099/ using document root <none> and password file <none>
I20260928 16:37:00.921958 9427 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:00.922051 9427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:00.922294 9427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:37:00.924422 9427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "0070eff24bcb423ca9ccfffbe4d34274"
format_stamp: "Formatted at 2026-09-28 16:37:00 on dist-test-slave-g9sh"
server_key: "3172d03e015b82200cc35a27338e7efb"
server_key_iv: "953289b24dbc04c3fb981d88a88524e5"
server_key_version: "encryptionkey@0"
I20260928 16:37:00.925060 9427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "0070eff24bcb423ca9ccfffbe4d34274"
format_stamp: "Formatted at 2026-09-28 16:37:00 on dist-test-slave-g9sh"
server_key: "3172d03e015b82200cc35a27338e7efb"
server_key_iv: "953289b24dbc04c3fb981d88a88524e5"
server_key_version: "encryptionkey@0"
I20260928 16:37:00.928707 9427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:37:00.931118 9442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:00.932310 9427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:37:00.932456 9427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "0070eff24bcb423ca9ccfffbe4d34274"
format_stamp: "Formatted at 2026-09-28 16:37:00 on dist-test-slave-g9sh"
server_key: "3172d03e015b82200cc35a27338e7efb"
server_key_iv: "953289b24dbc04c3fb981d88a88524e5"
server_key_version: "encryptionkey@0"
I20260928 16:37:00.932564 9427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:00.953918 9427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:00.957039 9427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:00.957322 9427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:00.964601 9427 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:45059
I20260928 16:37:00.964617 9500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:45059 every 8 connection(s)
I20260928 16:37:00.965669 9427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:37:00.968408 9501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:00.974207 9501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274: Bootstrap starting.
I20260928 16:37:00.975224 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 9427
I20260928 16:37:00.975354 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:37:00.975625 28139 external_mini_cluster.cc:1428] Setting key 1b58fa142b71a80a26e9700d19a454d1
I20260928 16:37:00.977087 9501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:00.977985 9501 log.cc:824] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:00.980208 9501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274: No bootstrap required, opened a new log
I20260928 16:37:00.984308 9501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0070eff24bcb423ca9ccfffbe4d34274" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 45059 } }
I20260928 16:37:00.984519 9501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:00.984599 9501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0070eff24bcb423ca9ccfffbe4d34274, State: Initialized, Role: FOLLOWER
Sep 28 16:37:00 dist-test-slave-g9sh krb5kdc[9413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:37:00.985129 9501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0070eff24bcb423ca9ccfffbe4d34274" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 45059 } }
I20260928 16:37:00.985275 9501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:37:00.985357 9501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:37:00.985459 9501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:00.986652 9501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0070eff24bcb423ca9ccfffbe4d34274" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 45059 } }
I20260928 16:37:00.987090 9501 leader_election.cc:302] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0070eff24bcb423ca9ccfffbe4d34274; no voters:
I20260928 16:37:00.987493 9501 leader_election.cc:288] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:37:00.987532 9506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:37:00.987787 9506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [term 1 LEADER]: Becoming Leader. State: Replica: 0070eff24bcb423ca9ccfffbe4d34274, State: Running, Role: LEADER
I20260928 16:37:00.988152 9506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0070eff24bcb423ca9ccfffbe4d34274" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 45059 } }
I20260928 16:37:00.988890 9501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:37:00.989866 9508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0070eff24bcb423ca9ccfffbe4d34274. Latest consensus state: current_term: 1 leader_uuid: "0070eff24bcb423ca9ccfffbe4d34274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0070eff24bcb423ca9ccfffbe4d34274" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 45059 } } }
I20260928 16:37:00.989986 9508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [sys.catalog]: This master's current role is: LEADER
I20260928 16:37:00.990358 9516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:37:00.990309 9507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0070eff24bcb423ca9ccfffbe4d34274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0070eff24bcb423ca9ccfffbe4d34274" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 45059 } } }
I20260928 16:37:00.990613 9507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274 [sys.catalog]: This master's current role is: LEADER
I20260928 16:37:00.991855 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:00.976727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.27.122.254:45059)
0928 16:37:00.977277 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:00.977298 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:00.977348 (+ 50us) server_negotiation.cc:407] Connection header received
0928 16:37:00.978217 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:00.978240 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:00.978623 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:00.978994 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:00.979683 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:00.981202 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:00.982041 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:00.982784 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:00.982977 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:00.985722 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:00.985728 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:00.985746 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:00.985783 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:00.988062 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:00.988625 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:00.988630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:00.988636 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:00.988775 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:00.989146 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:00.989150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:00.989152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:00.989716 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:00.989980 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:00.990614 (+ 634us) server_negotiation.cc:299] Negotiation successful
0928 16:37:00.990911 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":386,"thread_start_us":194,"threads_started":1}
I20260928 16:37:00.994074 9516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:37:00.994760 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:37:00.999411 9516 catalog_manager.cc:1347] Generated new cluster ID: fb401f7017674253abb694141f82f875
I20260928 16:37:00.999488 9516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:37:01.025826 9516 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:37:01.026839 9516 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:37:01.033417 9516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0070eff24bcb423ca9ccfffbe4d34274: Generated new TSK 0
I20260928 16:37:01.034132 9516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:37:01.050258 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 9427
2026-09-28T16:37:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1443 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:37:01 dist-test-slave-g9sh krb5kdc[9529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:37:01 dist-test-slave-g9sh krb5kdc[9529](info): set up 2 sockets
Sep 28 16:37:01 dist-test-slave-g9sh krb5kdc[9529](info): commencing operation
krb5kdc: starting...
W20260928 16:37:02.095593 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:37:02 dist-test-slave-g9sh krb5kdc[9529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:37:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:37:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:37:02.215109 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:42599
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:42599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:37:02.323392 9542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:02.323623 9542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:02.323669 9542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:02.327672 9542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:37:02.327741 9542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:02.327760 9542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:02.327777 9542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:37:02.327792 9542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:37:02.332842 9542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:42599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:42599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:02.333770 9542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:02.334697 9542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:02.343197 9548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:02.343226 9550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:02.343225 9547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:02.343366 9542 server_base.cc:1034] running on GCE node
I20260928 16:37:02.344033 9542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:02.345167 9542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:02.346330 9542 hybrid_clock.cc:648] HybridClock initialized: now 1790613422346316 us; error 27 us; skew 500 ppm
Sep 28 16:37:02 dist-test-slave-g9sh krb5kdc[9529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613422, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:02.349045 9542 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:37:02.349949 9542 webserver.cc:460] Webserver started at http://127.27.122.254:46435/ using document root <none> and password file <none>
I20260928 16:37:02.350487 9542 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:02.350548 9542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:02.350781 9542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:37:02.352742 9542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "0a7298c7f346434382f15b9cdb8d553c"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "65712945a085e206a61591a383fc2ada"
server_key_iv: "24e8913f337af7e4a67eff1dbd45f378"
server_key_version: "encryptionkey@0"
I20260928 16:37:02.353348 9542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "0a7298c7f346434382f15b9cdb8d553c"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "65712945a085e206a61591a383fc2ada"
server_key_iv: "24e8913f337af7e4a67eff1dbd45f378"
server_key_version: "encryptionkey@0"
I20260928 16:37:02.357043 9542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:37:02.359357 9557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:02.360550 9542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:37:02.360697 9542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "0a7298c7f346434382f15b9cdb8d553c"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "65712945a085e206a61591a383fc2ada"
server_key_iv: "24e8913f337af7e4a67eff1dbd45f378"
server_key_version: "encryptionkey@0"
I20260928 16:37:02.360792 9542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:02.379704 9542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:02.382807 9542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:02.383003 9542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:02.390381 9542 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:42599
I20260928 16:37:02.390723 9612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:42599 every 8 connection(s)
I20260928 16:37:02.391373 9542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:37:02.394104 9613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:02.399550 9613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Bootstrap starting.
I20260928 16:37:02.400556 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 9542
I20260928 16:37:02.400710 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:37:02.401032 28139 external_mini_cluster.cc:1428] Setting key 4f5b036f8aafc82c8c3fbb89a9d600f0
I20260928 16:37:02.402354 9613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:02.403102 9613 log.cc:824] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:02.405388 9613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: No bootstrap required, opened a new log
I20260928 16:37:02.409747 9613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } }
I20260928 16:37:02.410035 9613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:02.410132 9613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7298c7f346434382f15b9cdb8d553c, State: Initialized, Role: FOLLOWER
Sep 28 16:37:02 dist-test-slave-g9sh krb5kdc[9529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:37:02.410625 9613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } }
I20260928 16:37:02.410897 9613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:37:02.411001 9613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:37:02.411118 9613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:02.412241 9613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } }
I20260928 16:37:02.412596 9613 leader_election.cc:302] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a7298c7f346434382f15b9cdb8d553c; no voters:
I20260928 16:37:02.412947 9613 leader_election.cc:288] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:37:02.413021 9618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:37:02.413268 9618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 1 LEADER]: Becoming Leader. State: Replica: 0a7298c7f346434382f15b9cdb8d553c, State: Running, Role: LEADER
I20260928 16:37:02.413748 9618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } }
I20260928 16:37:02.414368 9613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:37:02.415696 9620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a7298c7f346434382f15b9cdb8d553c. Latest consensus state: current_term: 1 leader_uuid: "0a7298c7f346434382f15b9cdb8d553c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } } }
I20260928 16:37:02.415838 9620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: This master's current role is: LEADER
I20260928 16:37:02.415697 9619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a7298c7f346434382f15b9cdb8d553c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } } }
I20260928 16:37:02.416246 9619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: This master's current role is: LEADER
I20260928 16:37:02.416630 9628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:37:02.417519 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:02.402164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.27.122.254:42599)
0928 16:37:02.402634 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:02.402656 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:02.402690 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:37:02.403365 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:02.403384 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:02.403814 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:02.404167 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:02.405245 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:02.406419 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:02.407670 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:02.408159 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:02.408280 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:02.411372 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:02.411377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:02.411389 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:02.411416 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:02.413381 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:02.414015 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:02.414021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:02.414027 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:02.414184 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:02.414534 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:02.414538 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:02.414540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:02.415078 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:02.415285 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:02.416224 (+ 939us) server_negotiation.cc:299] Negotiation successful
0928 16:37:02.416572 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":154,"threads_started":1}
I20260928 16:37:02.420207 9628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:37:02.425284 9628 catalog_manager.cc:1347] Generated new cluster ID: ed94774bc5bd42238fcb1486add77cfe
I20260928 16:37:02.425348 9628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:37:02.440117 9628 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:37:02.441044 9628 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:37:02.446599 9628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Generated new TSK 0
I20260928 16:37:02.447391 9628 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:37:02.482180 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:42599
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:37:02.609977 9639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:02.610210 9639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:02.610250 9639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:02.614138 9639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:02.614187 9639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:02.614243 9639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:37:02.619433 9639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:02.620380 9639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:02.621349 9639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:02.630959 9645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:02.631014 9644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:02.630993 9647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:02.631063 9639 server_base.cc:1034] running on GCE node
I20260928 16:37:02.631561 9639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:02.632165 9639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:02.633350 9639 hybrid_clock.cc:648] HybridClock initialized: now 1790613422633335 us; error 26 us; skew 500 ppm
Sep 28 16:37:02 dist-test-slave-g9sh krb5kdc[9529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613422, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:02.636261 9639 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:37:02.637123 9639 webserver.cc:460] Webserver started at http://127.27.122.193:38789/ using document root <none> and password file <none>
I20260928 16:37:02.637619 9639 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:02.637684 9639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:02.637845 9639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:37:02.639771 9639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "c3077d0b67944e20b4338dd1fe0374e4"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "44d1c39437ece18cead9aef2377c648f"
server_key_iv: "2f663d4fc49246537c105428dbc8cb6a"
server_key_version: "encryptionkey@0"
I20260928 16:37:02.640259 9639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "c3077d0b67944e20b4338dd1fe0374e4"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "44d1c39437ece18cead9aef2377c648f"
server_key_iv: "2f663d4fc49246537c105428dbc8cb6a"
server_key_version: "encryptionkey@0"
I20260928 16:37:02.643752 9639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:37:02.646135 9654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:02.647298 9639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:37:02.647433 9639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "c3077d0b67944e20b4338dd1fe0374e4"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "44d1c39437ece18cead9aef2377c648f"
server_key_iv: "2f663d4fc49246537c105428dbc8cb6a"
server_key_version: "encryptionkey@0"
I20260928 16:37:02.647543 9639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:02.670360 9639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:02.673364 9639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:02.673569 9639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:02.674161 9639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:02.674984 9639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:37:02.675056 9639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:02.675128 9639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:37:02.675175 9639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:02.684207 9639 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:38281
I20260928 16:37:02.684237 9772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:38281 every 8 connection(s)
I20260928 16:37:02.685144 9639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:37:02.688766 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 9639
I20260928 16:37:02.688892 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:37:02.689240 28139 external_mini_cluster.cc:1428] Setting key 6efbe9be1dc6cba6c0f384d81d564ea5
Sep 28 16:37:02 dist-test-slave-g9sh krb5kdc[9529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613422, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:37:02.700062 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:02.686860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:34431 (local address 127.27.122.254:42599)
0928 16:37:02.686994 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:02.686998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:02.687815 (+ 817us) server_negotiation.cc:407] Connection header received
0928 16:37:02.689578 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:02.689581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:02.689659 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:02.689746 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:02.691326 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:02.691871 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:02.693307 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:02.693564 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:02.693679 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:02.696177 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:02.696181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:02.696183 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:02.696216 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:02.697857 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:02.698393 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:02.698397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:02.698399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:02.698486 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:02.698867 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:02.698872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:02.698874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:02.699136 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:02.699240 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:02.699750 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 16:37:02.699936 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:37:02.700819 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:02.687215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:34431)
0928 16:37:02.687662 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:02.687704 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:02.689320 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:02.690042 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:02.690051 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:02.690426 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:02.691165 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:02.691188 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:02.692004 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:02.692011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:02.693175 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:02.693185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:02.693716 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:02.693721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:02.693886 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:02.694572 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:02.694589 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:02.696011 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:02.697994 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:02.698000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:02.698014 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:02.698281 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:02.698588 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:02.698591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:02.698593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:02.698733 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:02.699251 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:02.699256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:02.699480 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 16:37:02.699659 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 16:37:02.699962 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
I20260928 16:37:02.702116 9774 heartbeater.cc:347] Connected to a master server at 127.27.122.254:42599
I20260928 16:37:02.702359 9774 heartbeater.cc:464] Registering TS with master...
I20260928 16:37:02.702833 9774 heartbeater.cc:511] Master 127.27.122.254:42599 requested a full tablet report, sending...
I20260928 16:37:02.704324 9575 ts_manager.cc:194] Registered new tserver with Master: c3077d0b67944e20b4338dd1fe0374e4 (127.27.122.193:38281)
I20260928 16:37:02.705523 9575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:34431
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:37:02.718222 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:42599
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:37:02.836388 9779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:02.836621 9779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:02.836663 9779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:02.840584 9779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:02.840636 9779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:02.840692 9779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:37:02.846242 9779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:02.847136 9779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:02.848053 9779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:02.857795 9787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:02.857800 9784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:02.857780 9785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:02.857980 9779 server_base.cc:1034] running on GCE node
I20260928 16:37:02.858445 9779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:02.858981 9779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:02.860152 9779 hybrid_clock.cc:648] HybridClock initialized: now 1790613422860137 us; error 26 us; skew 500 ppm
Sep 28 16:37:02 dist-test-slave-g9sh krb5kdc[9529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613422, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:02.862800 9779 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:37:02.863632 9779 webserver.cc:460] Webserver started at http://127.27.122.194:40413/ using document root <none> and password file <none>
I20260928 16:37:02.864233 9779 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:02.864323 9779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:02.864550 9779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:37:02.866623 9779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "409bc21e24964f16afb28d4abfe22b83"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "52bcf1a903a373db0e8eb58e7eb501e2"
server_key_iv: "a941009f791633fe2984fd6ae4c94027"
server_key_version: "encryptionkey@0"
I20260928 16:37:02.867165 9779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "409bc21e24964f16afb28d4abfe22b83"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "52bcf1a903a373db0e8eb58e7eb501e2"
server_key_iv: "a941009f791633fe2984fd6ae4c94027"
server_key_version: "encryptionkey@0"
I20260928 16:37:02.870771 9779 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 16:37:02.873054 9795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:02.874236 9779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:37:02.874369 9779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "409bc21e24964f16afb28d4abfe22b83"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "52bcf1a903a373db0e8eb58e7eb501e2"
server_key_iv: "a941009f791633fe2984fd6ae4c94027"
server_key_version: "encryptionkey@0"
I20260928 16:37:02.874476 9779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:02.888402 9779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:02.891479 9779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:02.891674 9779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:02.892230 9779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:02.893069 9779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:37:02.893136 9779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:02.893205 9779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:37:02.893247 9779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:02.902155 9779 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:41917
I20260928 16:37:02.902182 9910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:41917 every 8 connection(s)
I20260928 16:37:02.902997 9779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:37:02.903908 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 9779
I20260928 16:37:02.904053 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:37:02.904346 28139 external_mini_cluster.cc:1428] Setting key 7896db83298959f124a49fa4549f2bc8
Sep 28 16:37:02 dist-test-slave-g9sh krb5kdc[9529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613422, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
I20260928 16:37:02.919690 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:02.904979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:56885 (local address 127.27.122.254:42599)
0928 16:37:02.905135 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:02.905140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:02.906073 (+ 933us) server_negotiation.cc:407] Connection header received
0928 16:37:02.908100 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:02.908105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:02.908169 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:02.908267 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:02.909464 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:02.910028 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:02.911926 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:02.912161 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:02.912250 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:02.915308 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:02.915312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:02.915317 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:02.915350 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:02.917115 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:02.917931 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:02.917935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:02.917936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:02.918029 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:02.918420 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:02.918423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:02.918425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:02.918668 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:02.918775 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:02.919280 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 16:37:02.919425 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:37:02.920382 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:02.905380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:56885)
0928 16:37:02.905906 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:02.905963 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:02.907886 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:02.908414 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:02.908420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:02.908795 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:02.909329 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:02.909344 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:02.910168 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:02.910175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:02.911734 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:02.911750 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:02.912333 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:02.912340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:02.912587 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:02.913269 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:02.913293 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:02.915134 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:02.917467 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:02.917473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:02.917487 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:02.917816 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:02.918151 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:02.918154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:02.918156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:02.918292 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:02.918784 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:02.918790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:02.919012 (+ 222us) client_negotiation.cc:769] Sending connection context
0928 16:37:02.919184 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 16:37:02.919427 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":167,"threads_started":1}
I20260928 16:37:02.921660 9912 heartbeater.cc:347] Connected to a master server at 127.27.122.254:42599
I20260928 16:37:02.921916 9912 heartbeater.cc:464] Registering TS with master...
I20260928 16:37:02.922503 9912 heartbeater.cc:511] Master 127.27.122.254:42599 requested a full tablet report, sending...
I20260928 16:37:02.923661 9575 ts_manager.cc:194] Registered new tserver with Master: 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917)
I20260928 16:37:02.924237 9575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:56885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:37:02.935568 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:42599
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:37:03.044477 9917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:03.044718 9917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:03.044795 9917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:03.048775 9917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:03.048828 9917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:03.048952 9917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:37:03.054284 9917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:03.055254 9917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:03.056224 9917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:03.065519 9925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:03.065526 9922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:03.065665 9923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:03.065800 9917 server_base.cc:1034] running on GCE node
I20260928 16:37:03.066229 9917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:03.066819 9917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:03.067993 9917 hybrid_clock.cc:648] HybridClock initialized: now 1790613423067973 us; error 28 us; skew 500 ppm
Sep 28 16:37:03 dist-test-slave-g9sh krb5kdc[9529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:03.070710 9917 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:37:03.071564 9917 webserver.cc:460] Webserver started at http://127.27.122.195:39753/ using document root <none> and password file <none>
I20260928 16:37:03.072115 9917 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:03.072175 9917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:03.072405 9917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:37:03.074381 9917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "9ee50be9d9b040a3b1f11c8af268a660"
format_stamp: "Formatted at 2026-09-28 16:37:03 on dist-test-slave-g9sh"
server_key: "19d28a05aabf8420739334073e831024"
server_key_iv: "dea52b4e47019abcb5d8f8de2379e448"
server_key_version: "encryptionkey@0"
I20260928 16:37:03.074930 9917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "9ee50be9d9b040a3b1f11c8af268a660"
format_stamp: "Formatted at 2026-09-28 16:37:03 on dist-test-slave-g9sh"
server_key: "19d28a05aabf8420739334073e831024"
server_key_iv: "dea52b4e47019abcb5d8f8de2379e448"
server_key_version: "encryptionkey@0"
I20260928 16:37:03.078457 9917 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 16:37:03.080705 9932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:03.081907 9917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:37:03.082047 9917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "9ee50be9d9b040a3b1f11c8af268a660"
format_stamp: "Formatted at 2026-09-28 16:37:03 on dist-test-slave-g9sh"
server_key: "19d28a05aabf8420739334073e831024"
server_key_iv: "dea52b4e47019abcb5d8f8de2379e448"
server_key_version: "encryptionkey@0"
I20260928 16:37:03.082154 9917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:03.096454 9917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:03.099134 9917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:03.099339 9917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:03.099892 9917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:03.100703 9917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:37:03.100773 9917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:03.100839 9917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:37:03.100869 9917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:03.109712 9917 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:37303
I20260928 16:37:03.109740 10050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:37303 every 8 connection(s)
I20260928 16:37:03.110594 9917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:37:03.111232 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 9917
I20260928 16:37:03.111382 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:37:03.111670 28139 external_mini_cluster.cc:1428] Setting key 33f8a02f8095ae0a59b91e2d14a93a0e
Sep 28 16:37:03 dist-test-slave-g9sh krb5kdc[9529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:37:03.124392 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.112504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:44581 (local address 127.27.122.254:42599)
0928 16:37:03.113064 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:03.113068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:03.113792 (+ 724us) server_negotiation.cc:407] Connection header received
0928 16:37:03.115294 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:03.115297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:03.115388 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:03.115467 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:03.116643 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.117213 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.118314 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.118526 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.118605 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.121084 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:03.121087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:03.121089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:03.121114 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:03.122605 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:03.123081 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:03.123083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:03.123084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:03.123161 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:03.123455 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:03.123457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:03.123458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:03.123620 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:03.123704 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:03.124151 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 16:37:03.124271 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461}
I20260928 16:37:03.125433 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.113057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:44581)
0928 16:37:03.113640 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:03.113681 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:03.115109 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:03.115629 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:03.115636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:03.115977 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:03.116486 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.116509 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.117318 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.117321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.118208 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.118217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.118677 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.118681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.118832 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.119371 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:03.119388 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:03.120870 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:03.122725 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:03.122730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:03.122740 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:03.122996 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:03.123241 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:03.123243 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:03.123244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:03.123377 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:03.123721 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:03.123726 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:03.123946 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 16:37:03.124102 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 16:37:03.124437 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":151,"threads_started":1}
I20260928 16:37:03.126533 10052 heartbeater.cc:347] Connected to a master server at 127.27.122.254:42599
I20260928 16:37:03.126735 10052 heartbeater.cc:464] Registering TS with master...
I20260928 16:37:03.127158 10052 heartbeater.cc:511] Master 127.27.122.254:42599 requested a full tablet report, sending...
I20260928 16:37:03.128093 9575 ts_manager.cc:194] Registered new tserver with Master: 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303)
I20260928 16:37:03.128672 9575 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:44581
I20260928 16:37:03.137704 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:37:03.147089 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.139672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.27.122.254:42599)
0928 16:37:03.139834 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:03.139839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:03.140011 (+ 172us) server_negotiation.cc:407] Connection header received
0928 16:37:03.140167 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:03.140170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:03.140221 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:03.140310 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:03.141091 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.141634 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.142479 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.142706 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.142837 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.144146 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:03.144150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:03.144153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:03.144180 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:03.145388 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:03.145823 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:03.145827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:03.145829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:03.145911 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:03.146201 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:03.146204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:03.146206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:03.146416 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:03.146565 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:03.146802 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 16:37:03.146917 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:37:03.149792 9575 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:37:03.152993 9575 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:37:03.167533 10065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.161347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:41917 (local address 127.0.0.1:37020)
0928 16:37:03.161877 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:03.161891 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:03.161978 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:03.163134 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:03.163137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:03.163155 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:03.163228 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.163232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.164558 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.164561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.165851 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.165861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.166959 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.166963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.167075 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.167118 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 16:37:03.167247 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 16:37:03.167374 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":36,"thread_start_us":91,"threads_started":1}
I20260928 16:37:03.167569 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.162078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.27.122.194:41917)
0928 16:37:03.162672 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:03.162685 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:03.162707 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:37:03.162785 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:03.162792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:03.162973 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:03.163150 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:03.163310 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.164424 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.165982 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.166820 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.166951 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.167086 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:03.167280 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 16:37:03.167396 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":176,"threads_started":1}
I20260928 16:37:03.167768 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.161799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:37303 (local address 127.0.0.1:41052)
0928 16:37:03.163388 (+ 1589us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:03.163398 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:03.163499 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:03.163984 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:03.163987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:03.164006 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:03.164101 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.164107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.165467 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.165471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.166425 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.166433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.167356 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.167360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.167443 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.167469 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:37:03.167555 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:37:03.167636 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1541,"mutex_wait_us":604,"thread_start_us":112,"threads_started":1}
I20260928 16:37:03.168128 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.162053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.27.122.195:37303)
0928 16:37:03.162417 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:03.162428 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:03.163490 (+ 1062us) server_negotiation.cc:407] Connection header received
0928 16:37:03.163635 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:03.163640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:03.163848 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:03.164007 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:03.164199 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.165318 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.166536 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.167237 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.167352 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.167469 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:03.167579 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 16:37:03.167991 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":98,"threads_started":1}
I20260928 16:37:03.169123 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.161042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:38281 (local address 127.0.0.1:43040)
0928 16:37:03.161394 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:03.161440 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:03.161564 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:03.162545 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:03.162552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:03.162577 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:03.162717 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.162725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.164388 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.164393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.165979 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.165990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.168693 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.168696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.168795 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.168823 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:37:03.168923 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 16:37:03.168997 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":101,"thread_start_us":122,"threads_started":1}
I20260928 16:37:03.169201 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.161164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.27.122.193:38281)
0928 16:37:03.161990 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:03.162007 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:03.162032 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:37:03.162114 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:03.162123 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:03.162348 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:03.162871 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:03.163034 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.164262 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.166087 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.166835 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.166951 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.167086 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:03.168955 (+ 1869us) server_negotiation.cc:299] Negotiation successful
0928 16:37:03.169056 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":135,"threads_started":1}
I20260928 16:37:03.170660 9844 tablet_service.cc:1467] Processing CreateTablet for tablet b177aba64d5643b0a66fe5bd474e1f4c (DEFAULT_TABLE table=test-table [id=6b7362c6995c4e44b842ddac5ca047ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:37:03.170670 9983 tablet_service.cc:1467] Processing CreateTablet for tablet b177aba64d5643b0a66fe5bd474e1f4c (DEFAULT_TABLE table=test-table [id=6b7362c6995c4e44b842ddac5ca047ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:37:03.171720 9844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b177aba64d5643b0a66fe5bd474e1f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:03.171753 9983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b177aba64d5643b0a66fe5bd474e1f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:03.173962 9704 tablet_service.cc:1467] Processing CreateTablet for tablet b177aba64d5643b0a66fe5bd474e1f4c (DEFAULT_TABLE table=test-table [id=6b7362c6995c4e44b842ddac5ca047ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:37:03.174957 9704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b177aba64d5643b0a66fe5bd474e1f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:03.178813 10069 tablet_bootstrap.cc:492] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660: Bootstrap starting.
I20260928 16:37:03.181495 10070 tablet_bootstrap.cc:492] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Bootstrap starting.
I20260928 16:37:03.182430 10069 tablet_bootstrap.cc:654] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:03.183161 10069 log.cc:824] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:03.183706 10070 tablet_bootstrap.cc:654] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:03.183789 10071 tablet_bootstrap.cc:492] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83: Bootstrap starting.
I20260928 16:37:03.185003 10070 log.cc:824] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:03.185251 10069 tablet_bootstrap.cc:492] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660: No bootstrap required, opened a new log
I20260928 16:37:03.185444 10069 ts_tablet_manager.cc:1397] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:37:03.186412 10071 tablet_bootstrap.cc:654] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:03.186941 10070 tablet_bootstrap.cc:492] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: No bootstrap required, opened a new log
I20260928 16:37:03.187124 10070 ts_tablet_manager.cc:1397] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:37:03.187721 10071 log.cc:824] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:03.189383 10069 raft_consensus.cc:348] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } } peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } }
I20260928 16:37:03.189651 10069 raft_consensus.cc:374] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:03.189736 10069 raft_consensus.cc:729] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee50be9d9b040a3b1f11c8af268a660, State: Initialized, Role: FOLLOWER
I20260928 16:37:03.189999 10071 tablet_bootstrap.cc:492] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83: No bootstrap required, opened a new log
I20260928 16:37:03.190158 10071 ts_tablet_manager.cc:1397] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 16:37:03.190191 10069 consensus_queue.cc:260] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } } peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } }
I20260928 16:37:03.190399 10070 raft_consensus.cc:348] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } } peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } }
I20260928 16:37:03.190742 10070 raft_consensus.cc:374] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:03.190857 10070 raft_consensus.cc:729] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3077d0b67944e20b4338dd1fe0374e4, State: Initialized, Role: FOLLOWER
I20260928 16:37:03.190896 10052 heartbeater.cc:503] Master 127.27.122.254:42599 was elected leader, sending a full tablet report...
I20260928 16:37:03.191174 10069 ts_tablet_manager.cc:1428] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:37:03.191370 10070 consensus_queue.cc:260] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } } peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } }
I20260928 16:37:03.192013 9774 heartbeater.cc:503] Master 127.27.122.254:42599 was elected leader, sending a full tablet report...
I20260928 16:37:03.192360 10070 ts_tablet_manager.cc:1428] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:37:03.193696 10071 raft_consensus.cc:348] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } } peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } }
I20260928 16:37:03.193863 10071 raft_consensus.cc:374] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:03.193899 10071 raft_consensus.cc:729] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409bc21e24964f16afb28d4abfe22b83, State: Initialized, Role: FOLLOWER
I20260928 16:37:03.194468 10071 consensus_queue.cc:260] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } } peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } }
I20260928 16:37:03.195549 9912 heartbeater.cc:503] Master 127.27.122.254:42599 was elected leader, sending a full tablet report...
I20260928 16:37:03.195591 10071 ts_tablet_manager.cc:1428] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260928 16:37:03.361523 10053 tablet.cc:2369] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:37:03.381383 10077 raft_consensus.cc:482] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:37:03.381603 10077 raft_consensus.cc:504] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } } peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } }
I20260928 16:37:03.382777 10077 leader_election.cc:288] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3077d0b67944e20b4338dd1fe0374e4 (127.27.122.193:38281), 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303)
I20260928 16:37:03.387455 9914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.383086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:38281 (local address 127.27.122.194:38181)
0928 16:37:03.383245 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:03.383263 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:03.383375 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:03.383908 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:03.383911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:03.383929 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:03.384077 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.384084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.385143 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.385149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.386190 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.386201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.387102 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.387105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.387189 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.387210 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:37:03.387291 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 16:37:03.387345 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":82}
I20260928 16:37:03.387480 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.383168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:38181 (local address 127.27.122.193:38281)
0928 16:37:03.383320 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:03.383325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:03.383342 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:37:03.383717 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:03.383720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:03.383777 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:03.383878 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:03.384196 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.385019 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.386332 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.386983 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.387108 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.387164 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:03.387319 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 16:37:03.387382 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:37:03.388077 10066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.383562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:51373 (local address 127.27.122.195:37303)
0928 16:37:03.383690 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:03.383695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:03.383815 (+ 120us) server_negotiation.cc:407] Connection header received
0928 16:37:03.383935 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:03.383938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:03.383996 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:03.384054 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:03.384334 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.385073 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.386396 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.386990 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.387108 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.387162 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:03.387922 (+ 760us) server_negotiation.cc:299] Negotiation successful
0928 16:37:03.387988 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:37:03.388090 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.383420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:37303 (local address 127.27.122.194:51373)
0928 16:37:03.383728 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:03.383739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:03.383831 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:03.384086 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:03.384089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:03.384106 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:03.384233 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.384238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.385183 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.385185 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.386290 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.386307 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.387663 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.387670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.387770 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.387789 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 16:37:03.387887 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:37:03.387952 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260928 16:37:03.388144 9725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b177aba64d5643b0a66fe5bd474e1f4c" candidate_uuid: "409bc21e24964f16afb28d4abfe22b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3077d0b67944e20b4338dd1fe0374e4" is_pre_election: true
I20260928 16:37:03.388433 9725 raft_consensus.cc:2454] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 409bc21e24964f16afb28d4abfe22b83 in term 0.
I20260928 16:37:03.388952 9796 leader_election.cc:302] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409bc21e24964f16afb28d4abfe22b83, c3077d0b67944e20b4338dd1fe0374e4; no voters:
I20260928 16:37:03.389249 10077 raft_consensus.cc:2781] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:37:03.389253 10004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b177aba64d5643b0a66fe5bd474e1f4c" candidate_uuid: "409bc21e24964f16afb28d4abfe22b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee50be9d9b040a3b1f11c8af268a660" is_pre_election: true
I20260928 16:37:03.389339 10077 raft_consensus.cc:482] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:37:03.389372 10077 raft_consensus.cc:3037] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:03.389530 10004 raft_consensus.cc:2454] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 409bc21e24964f16afb28d4abfe22b83 in term 0.
I20260928 16:37:03.390635 10077 raft_consensus.cc:504] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } } peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } }
I20260928 16:37:03.391008 10077 leader_election.cc:288] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [CANDIDATE]: Term 1 election: Requested vote from peers c3077d0b67944e20b4338dd1fe0374e4 (127.27.122.193:38281), 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303)
I20260928 16:37:03.391355 9725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b177aba64d5643b0a66fe5bd474e1f4c" candidate_uuid: "409bc21e24964f16afb28d4abfe22b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3077d0b67944e20b4338dd1fe0374e4"
I20260928 16:37:03.391381 10004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b177aba64d5643b0a66fe5bd474e1f4c" candidate_uuid: "409bc21e24964f16afb28d4abfe22b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee50be9d9b040a3b1f11c8af268a660"
I20260928 16:37:03.391510 10004 raft_consensus.cc:3037] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:03.391501 9725 raft_consensus.cc:3037] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:03.392534 9725 raft_consensus.cc:2454] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 409bc21e24964f16afb28d4abfe22b83 in term 1.
I20260928 16:37:03.392715 10004 raft_consensus.cc:2454] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 409bc21e24964f16afb28d4abfe22b83 in term 1.
I20260928 16:37:03.392799 9796 leader_election.cc:302] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409bc21e24964f16afb28d4abfe22b83, c3077d0b67944e20b4338dd1fe0374e4; no voters:
I20260928 16:37:03.393083 10077 raft_consensus.cc:2781] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:37:03.393316 10077 raft_consensus.cc:686] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [term 1 LEADER]: Becoming Leader. State: Replica: 409bc21e24964f16afb28d4abfe22b83, State: Running, Role: LEADER
I20260928 16:37:03.393664 10077 consensus_queue.cc:237] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } } peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } }
I20260928 16:37:03.397094 9575 catalog_manager.cc:5495] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194). New cstate: current_term: 1 leader_uuid: "409bc21e24964f16afb28d4abfe22b83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } health_report { overall_health: HEALTHY } } }
W20260928 16:37:03.404031 9913 tablet.cc:2369] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:37:03.436534 9775 tablet.cc:2369] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:37:03.462584 10067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.458920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.27.122.194:41917)
0928 16:37:03.459067 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:03.459071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:03.459087 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:37:03.459233 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:03.459236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:03.459296 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:03.459412 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:37:03.459707 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.460225 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.461115 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.461337 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.461446 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.461779 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:37:03.462194 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:37:03.462321 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:03.462383 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 16:37:03.462437 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 16:37:03.468451 9725 raft_consensus.cc:1261] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Refusing update from remote peer 409bc21e24964f16afb28d4abfe22b83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:37:03.468494 10004 raft_consensus.cc:1261] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660 [term 1 FOLLOWER]: Refusing update from remote peer 409bc21e24964f16afb28d4abfe22b83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:37:03.469231 10079 consensus_queue.cc:1035] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:37:03.469403 10077 consensus_queue.cc:1035] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:37:03.486135 9575 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:37:03.486301 9575 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:37:03.488536 9575 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Sending DeleteTablet for 3 replicas of tablet b177aba64d5643b0a66fe5bd474e1f4c
I20260928 16:37:03.489359 9704 tablet_service.cc:1514] Processing DeleteTablet for tablet b177aba64d5643b0a66fe5bd474e1f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:37:03 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:43040
I20260928 16:37:03.489357 9844 tablet_service.cc:1514] Processing DeleteTablet for tablet b177aba64d5643b0a66fe5bd474e1f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:37:03 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:37020
I20260928 16:37:03.489423 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 9639
I20260928 16:37:03.490849 9983 tablet_service.cc:1514] Processing DeleteTablet for tablet b177aba64d5643b0a66fe5bd474e1f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:37:03 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:41052
I20260928 16:37:03.491133 10092 tablet_replica.cc:330] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83: stopping tablet replica
I20260928 16:37:03.491400 10092 raft_consensus.cc:2229] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:37:03.491745 10092 raft_consensus.cc:2258] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:37:03.492125 10093 tablet_replica.cc:330] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660: stopping tablet replica
I20260928 16:37:03.492460 10093 raft_consensus.cc:2229] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:37:03.492806 10093 raft_consensus.cc:2258] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:37:03.493427 10092 ts_tablet_manager.cc:1905] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:37:03.495594 10092 ts_tablet_manager.cc:1918] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:37:03.495714 10092 log.cc:1196] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/b177aba64d5643b0a66fe5bd474e1f4c
I20260928 16:37:03.496135 10092 ts_tablet_manager.cc:1939] T b177aba64d5643b0a66fe5bd474e1f4c P 409bc21e24964f16afb28d4abfe22b83: Deleting consensus metadata
I20260928 16:37:03.497004 10093 ts_tablet_manager.cc:1905] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:37:03.497367 9560 catalog_manager.cc:4841] TS 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917): tablet b177aba64d5643b0a66fe5bd474e1f4c (table test-table [id=6b7362c6995c4e44b842ddac5ca047ce]) successfully deleted
W20260928 16:37:03.498626 9558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:38281 (error 108)
I20260928 16:37:03.498853 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 9779
I20260928 16:37:03.500561 10063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.500265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:38281 (local address 127.0.0.1:43052)
0928 16:37:03.500408 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:03.500479 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:38281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 16:37:03.500725 9558 catalog_manager.cc:4584] TS c3077d0b67944e20b4338dd1fe0374e4 (127.27.122.193:38281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b177aba64d5643b0a66fe5bd474e1f4c: Network error: Client connection negotiation failed: client connection to 127.27.122.193:38281: connect: Connection refused (error 111)
I20260928 16:37:03.501753 10093 ts_tablet_manager.cc:1918] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:37:03.501834 10093 log.cc:1196] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/b177aba64d5643b0a66fe5bd474e1f4c
I20260928 16:37:03.502120 10093 ts_tablet_manager.cc:1939] T b177aba64d5643b0a66fe5bd474e1f4c P 9ee50be9d9b040a3b1f11c8af268a660: Deleting consensus metadata
I20260928 16:37:03.503155 9558 catalog_manager.cc:4841] TS 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303): tablet b177aba64d5643b0a66fe5bd474e1f4c (table test-table [id=6b7362c6995c4e44b842ddac5ca047ce]) successfully deleted
I20260928 16:37:03.505385 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 9917
I20260928 16:37:03.511116 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 9542
I20260928 16:37:03.517875 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:42599
--webserver_interface=127.27.122.254
--webserver_port=46435
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:42599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:37:03.627627 10095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:03.627876 10095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:03.627916 10095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:03.631860 10095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:37:03.631912 10095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:03.631932 10095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:03.631948 10095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:37:03.631963 10095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:37:03.637027 10095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:42599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:42599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=46435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:03.637967 10095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:03.639001 10095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:03.652967 10103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:03.652930 10102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:03.652976 10105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:03.653069 10095 server_base.cc:1034] running on GCE node
I20260928 16:37:03.653789 10095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:03.654861 10095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:03.656055 10095 hybrid_clock.cc:648] HybridClock initialized: now 1790613423656035 us; error 30 us; skew 500 ppm
Sep 28 16:37:03 dist-test-slave-g9sh krb5kdc[9529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:03.658825 10095 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:37:03.659745 10095 webserver.cc:460] Webserver started at http://127.27.122.254:46435/ using document root <none> and password file <none>
I20260928 16:37:03.660343 10095 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:03.660456 10095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:03.664361 10095 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:37:03.666584 10112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:03.667755 10095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:37:03.667898 10095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "0a7298c7f346434382f15b9cdb8d553c"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "65712945a085e206a61591a383fc2ada"
server_key_iv: "24e8913f337af7e4a67eff1dbd45f378"
server_key_version: "encryptionkey@0"
I20260928 16:37:03.668509 10095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:03.688304 10095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:03.691349 10095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:03.691555 10095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:03.698887 10095 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:42599
I20260928 16:37:03.699195 10165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:42599 every 8 connection(s)
I20260928 16:37:03.700028 10095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:37:03.704195 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 10095
I20260928 16:37:03.704533 10166 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:37:03.704672 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:38281
--local_ip_for_outbound_sockets=127.27.122.193
--tserver_master_addrs=127.27.122.254:42599
--webserver_port=38789
--webserver_interface=127.27.122.193
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:37:03.706307 10166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Bootstrap starting.
I20260928 16:37:03.730531 10166 log.cc:824] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:03.741369 10166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:37:03.741783 10166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Bootstrap complete.
I20260928 16:37:03.747172 10166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } }
I20260928 16:37:03.747664 10166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7298c7f346434382f15b9cdb8d553c, State: Initialized, Role: FOLLOWER
I20260928 16:37:03.748234 10166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } }
I20260928 16:37:03.748415 10166 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:37:03.748507 10166 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:37:03.748613 10166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:37:03.750694 10166 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } }
I20260928 16:37:03.751051 10166 leader_election.cc:302] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a7298c7f346434382f15b9cdb8d553c; no voters:
I20260928 16:37:03.751366 10166 leader_election.cc:288] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:37:03.751482 10170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:37:03.751719 10170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 2 LEADER]: Becoming Leader. State: Replica: 0a7298c7f346434382f15b9cdb8d553c, State: Running, Role: LEADER
I20260928 16:37:03.752085 10170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } }
I20260928 16:37:03.752533 10166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:37:03.753068 10170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a7298c7f346434382f15b9cdb8d553c. Latest consensus state: current_term: 2 leader_uuid: "0a7298c7f346434382f15b9cdb8d553c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } } }
I20260928 16:37:03.753270 10170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: This master's current role is: LEADER
I20260928 16:37:03.753453 10171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0a7298c7f346434382f15b9cdb8d553c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } } }
I20260928 16:37:03.753652 10171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: This master's current role is: LEADER
I20260928 16:37:03.753688 10175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:37:03.758461 10175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:37:03.759126 10175 catalog_manager.cc:1259] Loaded cluster ID: ed94774bc5bd42238fcb1486add77cfe
I20260928 16:37:03.759202 10175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:37:03.760543 10175 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 16:37:03.765262 10175 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 16:37:03.770231 10175 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:37:03.770881 10175 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Loaded TSK: 0
I20260928 16:37:03.772045 10175 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:37:03.834280 10168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:03.834506 10168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:03.834553 10168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:03.838603 10168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:03.838668 10168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:03.838763 10168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:37:03.843845 10168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:38281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=38789
--tserver_master_addrs=127.27.122.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:03.844751 10168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:03.845706 10168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:03.855800 10197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:03.855804 10194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:03.855813 10193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:03.856177 10168 server_base.cc:1034] running on GCE node
I20260928 16:37:03.856618 10168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:03.857416 10168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:03.858618 10168 hybrid_clock.cc:648] HybridClock initialized: now 1790613423858590 us; error 35 us; skew 500 ppm
Sep 28 16:37:03 dist-test-slave-g9sh krb5kdc[9529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:03.861822 10168 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:37:03.862696 10168 webserver.cc:460] Webserver started at http://127.27.122.193:38789/ using document root <none> and password file <none>
I20260928 16:37:03.863229 10168 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:03.863301 10168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:03.867185 10168 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 16:37:03.869485 10204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:03.870834 10168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:37:03.870996 10168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "c3077d0b67944e20b4338dd1fe0374e4"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "44d1c39437ece18cead9aef2377c648f"
server_key_iv: "2f663d4fc49246537c105428dbc8cb6a"
server_key_version: "encryptionkey@0"
I20260928 16:37:03.871650 10168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:03.880796 10168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:03.883929 10168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:03.884124 10168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:03.884688 10168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:03.885864 10211 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:37:03.889804 10168 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:37:03.889863 10168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260928 16:37:03.889930 10168 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:37:03.891569 10168 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:37:03.891624 10168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 16:37:03.891873 10211 tablet_bootstrap.cc:492] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Bootstrap starting.
I20260928 16:37:03.902850 10168 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:38281
I20260928 16:37:03.902864 10321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:38281 every 8 connection(s)
I20260928 16:37:03.904047 10168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:37:03.911788 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 10168
I20260928 16:37:03.912308 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:41917
--local_ip_for_outbound_sockets=127.27.122.194
--tserver_master_addrs=127.27.122.254:42599
--webserver_port=40413
--webserver_interface=127.27.122.194
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:37:03.912809 10211 log.cc:824] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:37:03 dist-test-slave-g9sh krb5kdc[9529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613423, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:37:03.921092 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.906193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:59567 (local address 127.27.122.254:42599)
0928 16:37:03.906555 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:03.906569 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:03.906968 (+ 399us) server_negotiation.cc:407] Connection header received
0928 16:37:03.908058 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:03.908068 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:03.908342 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:03.908684 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:03.909876 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.910841 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.912295 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.913055 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.913256 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.915814 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:03.915820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:03.915837 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:03.915872 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:03.917944 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:03.918466 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:03.918471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:03.918477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:03.918594 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:03.918931 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:03.918934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:03.918936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:03.919297 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:03.919446 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:03.919965 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 16:37:03.920234 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":129,"threads_started":1}
I20260928 16:37:03.921103 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.906234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:59567)
0928 16:37:03.906805 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:03.906845 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:03.907658 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:03.908742 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:03.908750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:03.909197 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:03.909691 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.909706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.911010 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.911014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.912155 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.912168 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.913214 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.913221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.913470 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.914190 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:03.914207 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:03.915616 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:03.918086 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:03.918092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:03.918104 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:03.918350 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:03.918694 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:03.918698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:03.918700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:03.918835 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:03.919418 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:03.919424 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:03.919669 (+ 245us) client_negotiation.cc:769] Sending connection context
0928 16:37:03.919852 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 16:37:03.920171 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":151,"threads_started":1}
I20260928 16:37:03.922477 10211 tablet_bootstrap.cc:492] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:37:03.922953 10211 tablet_bootstrap.cc:492] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Bootstrap complete.
I20260928 16:37:03.923748 10322 heartbeater.cc:347] Connected to a master server at 127.27.122.254:42599
I20260928 16:37:03.923972 10322 heartbeater.cc:464] Registering TS with master...
I20260928 16:37:03.924430 10322 heartbeater.cc:511] Master 127.27.122.254:42599 requested a full tablet report, sending...
I20260928 16:37:03.924525 10211 ts_tablet_manager.cc:1397] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.008s
I20260928 16:37:03.927081 10130 ts_manager.cc:194] Registered new tserver with Master: c3077d0b67944e20b4338dd1fe0374e4 (127.27.122.193:38281)
I20260928 16:37:03.927608 10211 raft_consensus.cc:348] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } } peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } }
I20260928 16:37:03.928107 10211 raft_consensus.cc:729] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3077d0b67944e20b4338dd1fe0374e4, State: Initialized, Role: FOLLOWER
I20260928 16:37:03.928570 10211 consensus_queue.cc:260] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } } peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } }
I20260928 16:37:03.929510 10211 ts_tablet_manager.cc:1428] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260928 16:37:03.930819 10130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:59567
I20260928 16:37:03.933180 10322 heartbeater.cc:503] Master 127.27.122.254:42599 was elected leader, sending a full tablet report...
I20260928 16:37:03.936575 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.930173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:38281 (local address 127.0.0.1:43062)
0928 16:37:03.931641 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:03.931671 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:03.931821 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:03.932618 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:03.932625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:03.932649 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:03.932760 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.932766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.934173 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.934177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.935191 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:03.935201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.936198 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.936201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:03.936288 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.936313 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 16:37:03.936390 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:37:03.936469 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"thread_start_us":109,"threads_started":1}
I20260928 16:37:03.936646 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:03.930264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43062 (local address 127.27.122.193:38281)
0928 16:37:03.930601 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:03.930612 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:03.931779 (+ 1167us) server_negotiation.cc:407] Connection header received
0928 16:37:03.931962 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:03.931968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:03.932429 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:03.932611 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:03.932882 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.934052 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.935306 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:03.936085 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:03.936207 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:03.936328 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:03.936432 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 16:37:03.936516 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":91,"threads_started":1}
I20260928 16:37:03.937587 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet b177aba64d5643b0a66fe5bd474e1f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:37:03 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:43062
I20260928 16:37:03.938352 10331 tablet_replica.cc:330] stopping tablet replica
I20260928 16:37:03.938659 10331 raft_consensus.cc:2229] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:37:03.938897 10331 raft_consensus.cc:2258] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:37:03.939781 10331 ts_tablet_manager.cc:1905] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:37:03.943190 10331 ts_tablet_manager.cc:1918] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:37:03.943301 10331 log.cc:1196] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal/wals/b177aba64d5643b0a66fe5bd474e1f4c
I20260928 16:37:03.943610 10331 ts_tablet_manager.cc:1939] T b177aba64d5643b0a66fe5bd474e1f4c P c3077d0b67944e20b4338dd1fe0374e4: Deleting consensus metadata
I20260928 16:37:03.945137 10113 catalog_manager.cc:4841] TS c3077d0b67944e20b4338dd1fe0374e4 (127.27.122.193:38281): tablet b177aba64d5643b0a66fe5bd474e1f4c (table test-table [id=6b7362c6995c4e44b842ddac5ca047ce]) successfully deleted
W20260928 16:37:04.029752 10327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:04.029970 10327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:04.030010 10327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:04.033975 10327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:04.034026 10327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:04.034102 10327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:37:04.039335 10327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:41917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=40413
--tserver_master_addrs=127.27.122.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:04.040287 10327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:04.041307 10327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:04.050369 10337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:04.050367 10340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:04.050374 10336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:04.050766 10327 server_base.cc:1034] running on GCE node
I20260928 16:37:04.051522 10327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:04.052039 10327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:04.053190 10327 hybrid_clock.cc:648] HybridClock initialized: now 1790613424053165 us; error 51 us; skew 500 ppm
Sep 28 16:37:04 dist-test-slave-g9sh krb5kdc[9529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613424, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:04.055810 10327 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:37:04.056540 10327 webserver.cc:460] Webserver started at http://127.27.122.194:40413/ using document root <none> and password file <none>
I20260928 16:37:04.057101 10327 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:04.057173 10327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:04.060691 10327 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 16:37:04.062759 10347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:04.063926 10327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:37:04.064056 10327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "409bc21e24964f16afb28d4abfe22b83"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "52bcf1a903a373db0e8eb58e7eb501e2"
server_key_iv: "a941009f791633fe2984fd6ae4c94027"
server_key_version: "encryptionkey@0"
I20260928 16:37:04.064571 10327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:04.081199 10327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:04.084215 10327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:04.084370 10327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:04.085072 10327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:04.085912 10327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:37:04.085960 10327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:04.086025 10327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:37:04.086058 10327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:04.094799 10327 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:41917
I20260928 16:37:04.094873 10461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:41917 every 8 connection(s)
I20260928 16:37:04.095839 10327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:37:04.100602 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 10327
I20260928 16:37:04.101056 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:37303
--local_ip_for_outbound_sockets=127.27.122.195
--tserver_master_addrs=127.27.122.254:42599
--webserver_port=39753
--webserver_interface=127.27.122.195
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:37:04 dist-test-slave-g9sh krb5kdc[9529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613424, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:37:04.110649 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.097755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:59819 (local address 127.27.122.254:42599)
0928 16:37:04.097899 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:04.097903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:04.098748 (+ 845us) server_negotiation.cc:407] Connection header received
0928 16:37:04.100130 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:04.100134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:04.100201 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:04.100307 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:04.101670 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.102274 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.103461 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.103902 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.104030 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.106514 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:04.106520 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:04.106526 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:04.106563 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:04.108414 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:04.108895 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:04.108918 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:04.108922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:04.109025 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:04.109352 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:04.109356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:04.109360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:04.109631 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:04.109765 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:04.110233 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 16:37:04.110395 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:37:04.111261 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.098165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:59819)
0928 16:37:04.098597 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:04.098641 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:04.099910 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:04.100506 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:04.100513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:04.100963 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:04.101517 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:04.101532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.102429 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.102432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:04.103327 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:04.103337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.104062 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.104064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:04.104208 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.104852 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:04.104878 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:04.106337 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:04.108536 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:04.108541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:04.108555 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:04.108793 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:04.109123 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:04.109125 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:04.109126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:04.109253 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:04.109754 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:04.109759 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:04.109983 (+ 224us) client_negotiation.cc:769] Sending connection context
0928 16:37:04.110142 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 16:37:04.110383 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":100,"threads_started":1}
I20260928 16:37:04.112419 10462 heartbeater.cc:347] Connected to a master server at 127.27.122.254:42599
I20260928 16:37:04.112676 10462 heartbeater.cc:464] Registering TS with master...
I20260928 16:37:04.113201 10462 heartbeater.cc:511] Master 127.27.122.254:42599 requested a full tablet report, sending...
I20260928 16:37:04.114245 10130 ts_manager.cc:194] Registered new tserver with Master: 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917)
I20260928 16:37:04.114815 10130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:59819
W20260928 16:37:04.214146 10465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:04.214423 10465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:04.214493 10465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:04.218659 10465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:04.218737 10465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:04.218827 10465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:37:04.223893 10465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:37303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=39753
--tserver_master_addrs=127.27.122.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:04.224879 10465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:04.225922 10465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:04.235139 10473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:04.235152 10471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:04.235154 10470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:04.235653 10465 server_base.cc:1034] running on GCE node
I20260928 16:37:04.236040 10465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:04.236657 10465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:04.237820 10465 hybrid_clock.cc:648] HybridClock initialized: now 1790613424237774 us; error 59 us; skew 500 ppm
Sep 28 16:37:04 dist-test-slave-g9sh krb5kdc[9529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613424, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:04.240554 10465 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:37:04.241444 10465 webserver.cc:460] Webserver started at http://127.27.122.195:39753/ using document root <none> and password file <none>
I20260928 16:37:04.241986 10465 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:04.242086 10465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:04.245790 10465 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:37:04.247781 10480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:04.249017 10465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:37:04.249135 10465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "9ee50be9d9b040a3b1f11c8af268a660"
format_stamp: "Formatted at 2026-09-28 16:37:03 on dist-test-slave-g9sh"
server_key: "19d28a05aabf8420739334073e831024"
server_key_iv: "dea52b4e47019abcb5d8f8de2379e448"
server_key_version: "encryptionkey@0"
I20260928 16:37:04.249656 10465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:04.260355 10465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:04.263336 10465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:04.263556 10465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:04.264117 10465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:04.265007 10465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:37:04.265069 10465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:04.265137 10465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:37:04.265178 10465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:04.274680 10465 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:37303
I20260928 16:37:04.274708 10598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:37303 every 8 connection(s)
I20260928 16:37:04.275730 10465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:37:04.277534 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 10465
Sep 28 16:37:04 dist-test-slave-g9sh krb5kdc[9529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613424, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:37:04.286751 10602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.278224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.27.122.254:42599)
0928 16:37:04.278555 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:04.278559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:04.278575 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:37:04.278726 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:04.278730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:04.278784 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:04.278885 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:04.279745 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.280551 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.281472 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.281867 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.281944 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.283362 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:04.283366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:04.283369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:04.283395 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:04.285118 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:04.285523 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:04.285527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:04.285529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:04.285617 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:04.285892 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:04.285896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:04.285897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:04.286071 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:04.286217 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:04.286461 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 16:37:04.286609 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":244,"thread_start_us":84,"threads_started":1}
I20260928 16:37:04.290531 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.277598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:42525 (local address 127.27.122.254:42599)
0928 16:37:04.277736 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:04.277739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:04.278833 (+ 1094us) server_negotiation.cc:407] Connection header received
0928 16:37:04.280398 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:04.280401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:04.280437 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:04.280497 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:04.281817 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.282583 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.283955 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.284206 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.284318 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.287184 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:04.287186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:04.287188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:04.287208 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:04.288770 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:04.289274 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:04.289277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:04.289278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:04.289348 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:04.289665 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:04.289668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:04.289669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:04.289817 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:04.289888 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:04.290336 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 16:37:04.290423 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":53}
I20260928 16:37:04.291361 10603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.278108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:42525)
0928 16:37:04.278674 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:04.278726 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:04.280181 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:04.280677 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:04.280686 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:04.281109 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:04.281676 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:04.281694 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.282717 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.282720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:04.283826 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:04.283842 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.284384 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.284391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:04.284634 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.285204 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:04.285228 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:04.287022 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:04.288870 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:04.288874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:04.288885 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:04.289189 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:04.289429 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:04.289431 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:04.289433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:04.289578 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:04.289913 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:04.289918 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:04.290131 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 16:37:04.290274 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 16:37:04.290510 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":170,"threads_started":1}
I20260928 16:37:04.292434 10599 heartbeater.cc:347] Connected to a master server at 127.27.122.254:42599
I20260928 16:37:04.292687 10599 heartbeater.cc:464] Registering TS with master...
I20260928 16:37:04.293233 10599 heartbeater.cc:511] Master 127.27.122.254:42599 requested a full tablet report, sending...
I20260928 16:37:04.294202 10130 ts_manager.cc:194] Registered new tserver with Master: 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303)
I20260928 16:37:04.294757 10130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:42525
I20260928 16:37:04.300088 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:37:04.303840 10324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.300679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.27.122.254:42599)
0928 16:37:04.300795 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:04.300798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:04.300997 (+ 199us) server_negotiation.cc:407] Connection header received
0928 16:37:04.301126 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:04.301128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:04.301212 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:04.301292 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:37:04.301467 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.301936 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.302730 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.302927 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.303000 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.303247 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:37:04.303573 (+ 326us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:37:04.303665 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:04.303713 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 16:37:04.303752 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:37:04.304399 10130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:37:04.307583 10130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:37:04.315316 10254 tablet_service.cc:1467] Processing CreateTablet for tablet c8e3d761036b484286202fe4d56681b8 (DEFAULT_TABLE table=test-table [id=622e3acfc28e4553aaa40a0bd4491673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:37:04.316007 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e3d761036b484286202fe4d56681b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:04.318876 10211 tablet_bootstrap.cc:492] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4: Bootstrap starting.
I20260928 16:37:04.318967 10329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.314288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:37303 (local address 127.0.0.1:41064)
0928 16:37:04.314442 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:04.314457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:04.314537 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:04.315340 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:04.315343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:04.315362 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:04.315471 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:04.315481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.316707 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.316714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:04.317763 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:04.317772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.318570 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.318582 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:04.318666 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.318687 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:37:04.318774 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 16:37:04.318850 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 16:37:04.319015 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.314417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.27.122.195:37303)
0928 16:37:04.314850 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:04.314862 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:04.314885 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:37:04.314956 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:04.314962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:04.315187 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:04.315371 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:04.315628 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.316579 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.317884 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.318442 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.318577 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.318706 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:04.318801 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 16:37:04.318888 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":146,"threads_started":1}
I20260928 16:37:04.319751 10211 tablet_bootstrap.cc:654] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:04.321122 10609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.314503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:41917 (local address 127.0.0.1:37048)
0928 16:37:04.314831 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:04.314846 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:04.314930 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:04.315794 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:04.315797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:04.315821 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:04.315901 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:04.315907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.317323 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.317328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:04.318534 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:04.318549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.320689 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.320693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:04.320791 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.320814 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:37:04.320891 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:37:04.321000 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":101,"threads_started":1}
I20260928 16:37:04.321223 10211 tablet_bootstrap.cc:492] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4: No bootstrap required, opened a new log
I20260928 16:37:04.321313 10211 ts_tablet_manager.cc:1397] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 16:37:04.321405 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.314917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.27.122.194:41917)
0928 16:37:04.315279 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:04.315290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:04.315314 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:37:04.315391 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:04.315397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:04.315642 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:04.315806 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:04.315987 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.317047 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.318705 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.319513 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.319653 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.319802 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:04.320964 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0928 16:37:04.321071 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":125,"threads_started":1}
I20260928 16:37:04.321787 10211 raft_consensus.cc:348] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:04.321908 10211 raft_consensus.cc:374] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:04.321954 10211 raft_consensus.cc:729] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3077d0b67944e20b4338dd1fe0374e4, State: Initialized, Role: FOLLOWER
I20260928 16:37:04.322082 10211 consensus_queue.cc:260] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:04.322459 10211 ts_tablet_manager.cc:1428] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 16:37:04.322242 10529 tablet_service.cc:1467] Processing CreateTablet for tablet c8e3d761036b484286202fe4d56681b8 (DEFAULT_TABLE table=test-table [id=622e3acfc28e4553aaa40a0bd4491673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:37:04.323304 10529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e3d761036b484286202fe4d56681b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:04.324281 10396 tablet_service.cc:1467] Processing CreateTablet for tablet c8e3d761036b484286202fe4d56681b8 (DEFAULT_TABLE table=test-table [id=622e3acfc28e4553aaa40a0bd4491673]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:37:04.325347 10396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e3d761036b484286202fe4d56681b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:04.330363 10612 tablet_bootstrap.cc:492] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660: Bootstrap starting.
I20260928 16:37:04.331822 10613 tablet_bootstrap.cc:492] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83: Bootstrap starting.
I20260928 16:37:04.333309 10612 tablet_bootstrap.cc:654] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:04.334026 10612 log.cc:824] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:04.334369 10613 tablet_bootstrap.cc:654] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:04.335129 10613 log.cc:824] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:04.335839 10612 tablet_bootstrap.cc:492] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660: No bootstrap required, opened a new log
I20260928 16:37:04.336035 10612 ts_tablet_manager.cc:1397] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:37:04.336763 10613 tablet_bootstrap.cc:492] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83: No bootstrap required, opened a new log
I20260928 16:37:04.336971 10613 ts_tablet_manager.cc:1397] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:37:04.339315 10612 raft_consensus.cc:348] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:04.339542 10612 raft_consensus.cc:374] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:04.339607 10612 raft_consensus.cc:729] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee50be9d9b040a3b1f11c8af268a660, State: Initialized, Role: FOLLOWER
I20260928 16:37:04.340063 10612 consensus_queue.cc:260] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:04.340274 10613 raft_consensus.cc:348] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:04.340492 10613 raft_consensus.cc:374] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:04.340560 10613 raft_consensus.cc:729] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 409bc21e24964f16afb28d4abfe22b83, State: Initialized, Role: FOLLOWER
I20260928 16:37:04.340662 10599 heartbeater.cc:503] Master 127.27.122.254:42599 was elected leader, sending a full tablet report...
I20260928 16:37:04.341056 10612 ts_tablet_manager.cc:1428] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260928 16:37:04.341012 10613 consensus_queue.cc:260] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:04.341646 10462 heartbeater.cc:503] Master 127.27.122.254:42599 was elected leader, sending a full tablet report...
I20260928 16:37:04.341936 10613 ts_tablet_manager.cc:1428] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260928 16:37:04.346740 10463 tablet.cc:2369] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:37:04.405300 10323 tablet.cc:2369] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:37:04.422422 10617 raft_consensus.cc:482] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:37:04.422665 10617 raft_consensus.cc:504] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:04.423935 10617 leader_election.cc:288] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303), c3077d0b67944e20b4338dd1fe0374e4 (127.27.122.193:38281)
I20260928 16:37:04.428007 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.424127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:37303 (local address 127.27.122.194:44949)
0928 16:37:04.424281 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:04.424318 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:04.424422 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:04.424750 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:04.424754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:04.424775 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:04.424891 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:04.424948 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.425875 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.425879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:04.426822 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:04.426832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.427562 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.427566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:04.427676 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.427702 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:37:04.427785 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 16:37:04.427857 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":21}
I20260928 16:37:04.428030 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.424261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:44949 (local address 127.27.122.195:37303)
0928 16:37:04.424417 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:04.424422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:04.424437 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:37:04.424516 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:04.424519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:04.424570 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:04.424686 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:04.425073 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.425763 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.426956 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.427442 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.427536 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.427577 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:04.427824 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 16:37:04.427898 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:37:04.428293 10618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.424484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:38281 (local address 127.27.122.194:44295)
0928 16:37:04.424844 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:04.424855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:04.425018 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:04.425272 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:04.425275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:04.425295 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:04.425406 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:04.425412 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.426202 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.426204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:04.427064 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:04.427074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.427839 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.427843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:04.427938 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.427962 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:37:04.428042 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 16:37:04.428115 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":22,"thread_start_us":117,"threads_started":1}
I20260928 16:37:04.428304 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.424599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:44295 (local address 127.27.122.193:38281)
0928 16:37:04.424726 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:04.424730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:04.424941 (+ 211us) server_negotiation.cc:407] Connection header received
0928 16:37:04.425133 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:04.425135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:04.425177 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:04.425238 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:04.425514 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.426090 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.427203 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.427710 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.427819 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.427861 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:04.428080 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 16:37:04.428168 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:37:04.428737 10550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e3d761036b484286202fe4d56681b8" candidate_uuid: "409bc21e24964f16afb28d4abfe22b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee50be9d9b040a3b1f11c8af268a660" is_pre_election: true
I20260928 16:37:04.428844 10275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e3d761036b484286202fe4d56681b8" candidate_uuid: "409bc21e24964f16afb28d4abfe22b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3077d0b67944e20b4338dd1fe0374e4" is_pre_election: true
I20260928 16:37:04.429077 10550 raft_consensus.cc:2454] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 409bc21e24964f16afb28d4abfe22b83 in term 0.
I20260928 16:37:04.429211 10275 raft_consensus.cc:2454] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 409bc21e24964f16afb28d4abfe22b83 in term 0.
I20260928 16:37:04.429607 10348 leader_election.cc:302] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409bc21e24964f16afb28d4abfe22b83, 9ee50be9d9b040a3b1f11c8af268a660; no voters:
I20260928 16:37:04.429859 10617 raft_consensus.cc:2781] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:37:04.429945 10617 raft_consensus.cc:482] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:37:04.429989 10617 raft_consensus.cc:3037] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:04.431012 10617 raft_consensus.cc:504] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:04.431361 10617 leader_election.cc:288] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [CANDIDATE]: Term 1 election: Requested vote from peers 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303), c3077d0b67944e20b4338dd1fe0374e4 (127.27.122.193:38281)
I20260928 16:37:04.431658 10550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e3d761036b484286202fe4d56681b8" candidate_uuid: "409bc21e24964f16afb28d4abfe22b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ee50be9d9b040a3b1f11c8af268a660"
I20260928 16:37:04.431694 10275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e3d761036b484286202fe4d56681b8" candidate_uuid: "409bc21e24964f16afb28d4abfe22b83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3077d0b67944e20b4338dd1fe0374e4"
I20260928 16:37:04.431792 10550 raft_consensus.cc:3037] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:04.431798 10275 raft_consensus.cc:3037] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:04.432833 10275 raft_consensus.cc:2454] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 409bc21e24964f16afb28d4abfe22b83 in term 1.
I20260928 16:37:04.432991 10550 raft_consensus.cc:2454] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 409bc21e24964f16afb28d4abfe22b83 in term 1.
I20260928 16:37:04.433221 10348 leader_election.cc:302] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 409bc21e24964f16afb28d4abfe22b83, c3077d0b67944e20b4338dd1fe0374e4; no voters:
I20260928 16:37:04.433403 10617 raft_consensus.cc:2781] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:37:04.433648 10617 raft_consensus.cc:686] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [term 1 LEADER]: Becoming Leader. State: Replica: 409bc21e24964f16afb28d4abfe22b83, State: Running, Role: LEADER
I20260928 16:37:04.433957 10617 consensus_queue.cc:237] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:04.437294 10129 catalog_manager.cc:5495] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194). New cstate: current_term: 1 leader_uuid: "409bc21e24964f16afb28d4abfe22b83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } health_report { overall_health: UNKNOWN } } }
I20260928 16:37:04.472316 10610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.468865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37062 (local address 127.27.122.194:41917)
0928 16:37:04.469024 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:04.469028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:04.469055 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:37:04.469224 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:04.469226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:04.469284 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:04.469374 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:37:04.469582 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.470060 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.470900 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:04.471099 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:04.471186 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:04.471469 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:37:04.471931 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:37:04.472021 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:04.472094 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 16:37:04.472150 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 16:37:04.478291 10550 raft_consensus.cc:1261] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660 [term 1 FOLLOWER]: Refusing update from remote peer 409bc21e24964f16afb28d4abfe22b83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:37:04.478305 10275 raft_consensus.cc:1261] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Refusing update from remote peer 409bc21e24964f16afb28d4abfe22b83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:37:04.479023 10619 consensus_queue.cc:1035] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:37:04.479202 10617 consensus_queue.cc:1035] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:37:04.486732 10622 mvcc.cc:205] Tried to move back new op lower bound from 7334352586655985664 to 7334352586484355072. Current Snapshot: MvccSnapshot[applied={T|T < 7334352586655985664 or (T in {7334352586655985664})}]
I20260928 16:37:04.487599 10623 mvcc.cc:205] Tried to move back new op lower bound from 7334352586655985664 to 7334352586484355072. Current Snapshot: MvccSnapshot[applied={T|T < 7334352586655985664}]
I20260928 16:37:04.493371 10129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:37:04.493541 10129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:37:04.495920 10129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Sending DeleteTablet for 3 replicas of tablet c8e3d761036b484286202fe4d56681b8
I20260928 16:37:04.496585 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e3d761036b484286202fe4d56681b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:37:04 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:43062
I20260928 16:37:04.496724 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 10168
I20260928 16:37:04.496665 10396 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e3d761036b484286202fe4d56681b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:37:04 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:37048
I20260928 16:37:04.496686 10529 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e3d761036b484286202fe4d56681b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:37:04 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:41064
I20260928 16:37:04.497460 10632 tablet_replica.cc:330] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83: stopping tablet replica
I20260928 16:37:04.497780 10632 raft_consensus.cc:2229] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:37:04.498162 10632 raft_consensus.cc:2258] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:37:04.499735 10632 ts_tablet_manager.cc:1905] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:37:04.501132 10633 tablet_replica.cc:330] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660: stopping tablet replica
I20260928 16:37:04.501494 10633 raft_consensus.cc:2229] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:37:04.501863 10633 raft_consensus.cc:2258] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:37:04.502652 10632 ts_tablet_manager.cc:1918] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:37:04.502758 10632 log.cc:1196] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/c8e3d761036b484286202fe4d56681b8
I20260928 16:37:04.503113 10632 ts_tablet_manager.cc:1939] T c8e3d761036b484286202fe4d56681b8 P 409bc21e24964f16afb28d4abfe22b83: Deleting consensus metadata
I20260928 16:37:04.503690 10633 ts_tablet_manager.cc:1905] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:37:04.504330 10115 catalog_manager.cc:4841] TS 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917): tablet c8e3d761036b484286202fe4d56681b8 (table test-table [id=622e3acfc28e4553aaa40a0bd4491673]) successfully deleted
W20260928 16:37:04.504715 10113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:38281 (error 108)
I20260928 16:37:04.505285 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 10327
I20260928 16:37:04.505848 10609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:04.505566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:38281 (local address 127.0.0.1:43068)
0928 16:37:04.505678 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:04.505756 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:38281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 16:37:04.506095 10113 catalog_manager.cc:4584] TS c3077d0b67944e20b4338dd1fe0374e4 (127.27.122.193:38281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8e3d761036b484286202fe4d56681b8: Network error: Client connection negotiation failed: client connection to 127.27.122.193:38281: connect: Connection refused (error 111)
I20260928 16:37:04.508318 10633 ts_tablet_manager.cc:1918] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:37:04.508442 10633 log.cc:1196] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/c8e3d761036b484286202fe4d56681b8
I20260928 16:37:04.508812 10633 ts_tablet_manager.cc:1939] T c8e3d761036b484286202fe4d56681b8 P 9ee50be9d9b040a3b1f11c8af268a660: Deleting consensus metadata
I20260928 16:37:04.509902 10113 catalog_manager.cc:4841] TS 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303): tablet c8e3d761036b484286202fe4d56681b8 (table test-table [id=622e3acfc28e4553aaa40a0bd4491673]) successfully deleted
I20260928 16:37:04.511816 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 10465
I20260928 16:37:04.519876 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 10095
I20260928 16:37:04.526363 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:42599
--webserver_interface=127.27.122.254
--webserver_port=46435
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:42599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:37:04.633607 10635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:04.633834 10635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:04.633872 10635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:04.637794 10635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:37:04.637841 10635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:04.637892 10635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:04.637926 10635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:37:04.637946 10635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:37:04.643114 10635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:42599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:42599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=46435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:04.644060 10635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:04.645044 10635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:37:04.653129 10635 server_base.cc:1034] running on GCE node
W20260928 16:37:04.653085 10641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:04.653111 10640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:04.653086 10643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:04.653913 10635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:04.654911 10635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:04.656117 10635 hybrid_clock.cc:648] HybridClock initialized: now 1790613424656080 us; error 47 us; skew 500 ppm
Sep 28 16:37:04 dist-test-slave-g9sh krb5kdc[9529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613424, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:04.658679 10635 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:37:04.659538 10635 webserver.cc:460] Webserver started at http://127.27.122.254:46435/ using document root <none> and password file <none>
I20260928 16:37:04.660071 10635 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:04.660159 10635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:04.663728 10635 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:37:04.665740 10651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:04.666850 10635 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:37:04.666988 10635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "0a7298c7f346434382f15b9cdb8d553c"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "65712945a085e206a61591a383fc2ada"
server_key_iv: "24e8913f337af7e4a67eff1dbd45f378"
server_key_version: "encryptionkey@0"
I20260928 16:37:04.667508 10635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:04.679710 10635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:04.682310 10635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:04.682493 10635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:04.689180 10635 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:42599
I20260928 16:37:04.689453 10705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:42599 every 8 connection(s)
I20260928 16:37:04.690210 10635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:37:04.692143 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 10635
I20260928 16:37:04.692633 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:38281
--local_ip_for_outbound_sockets=127.27.122.193
--tserver_master_addrs=127.27.122.254:42599
--webserver_port=38789
--webserver_interface=127.27.122.193
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:37:04.695114 10706 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:37:04.696769 10706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Bootstrap starting.
I20260928 16:37:04.714064 10706 log.cc:824] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:04.725349 10706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:37:04.725710 10706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c: Bootstrap complete.
I20260928 16:37:04.729727 10706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } }
I20260928 16:37:04.730130 10706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7298c7f346434382f15b9cdb8d553c, State: Initialized, Role: FOLLOWER
I20260928 16:37:04.730654 10706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } }
I20260928 16:37:04.730800 10706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:37:04.730890 10706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:37:04.731005 10706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:37:04.732796 10706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } }
I20260928 16:37:04.733146 10706 leader_election.cc:302] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a7298c7f346434382f15b9cdb8d553c; no voters:
I20260928 16:37:04.733479 10706 leader_election.cc:288] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 16:37:04.733559 10710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 3 FOLLOWER]: Leader election won for term 3
I20260928 16:37:04.733759 10710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [term 3 LEADER]: Becoming Leader. State: Replica: 0a7298c7f346434382f15b9cdb8d553c, State: Running, Role: LEADER
I20260928 16:37:04.734041 10710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } }
I20260928 16:37:04.734686 10706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:37:04.734937 10712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0a7298c7f346434382f15b9cdb8d553c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } } }
I20260928 16:37:04.735656 10712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: This master's current role is: LEADER
I20260928 16:37:04.735541 10711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a7298c7f346434382f15b9cdb8d553c. Latest consensus state: current_term: 3 leader_uuid: "0a7298c7f346434382f15b9cdb8d553c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7298c7f346434382f15b9cdb8d553c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 42599 } } }
I20260928 16:37:04.735972 10711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7298c7f346434382f15b9cdb8d553c [sys.catalog]: This master's current role is: LEADER
I20260928 16:37:04.736394 10720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:37:04.741410 10720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:37:04.742076 10720 catalog_manager.cc:1259] Loaded cluster ID: ed94774bc5bd42238fcb1486add77cfe
I20260928 16:37:04.742148 10720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 16:37:04.743254 10720 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f92861439ad google::LogMessage::Fail() at ??:0
@ 0x7f928614688a google::LogMessage::SendToLog() at ??:0
@ 0x7f92861434de google::LogMessage::Flush() at ??:0
@ 0x7f9286144a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9290a86fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9290a87d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9290a84579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9290ac8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f92883953d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9288395c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f92883973f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9288386512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f92899ca6db start_thread at ??:0
@ 0x7f9284c4161f clone at ??:0
W20260928 16:37:04.813274 10708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:04.813482 10708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:04.813522 10708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:04.817731 10708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:04.817770 10708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:04.817822 10708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:37:04.823297 10708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:38281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=38789
--tserver_master_addrs=127.27.122.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:04.824280 10708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:04.825313 10708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:04.850008 10748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:04.850034 10751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:04.850219 10708 server_base.cc:1034] running on GCE node
W20260928 16:37:04.850492 10749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:04.850934 10708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:04.851672 10708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:04.852854 10708 hybrid_clock.cc:648] HybridClock initialized: now 1790613424852819 us; error 48 us; skew 500 ppm
Sep 28 16:37:04 dist-test-slave-g9sh krb5kdc[9529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613424, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:04.855594 10708 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:37:04.856508 10708 webserver.cc:460] Webserver started at http://127.27.122.193:38789/ using document root <none> and password file <none>
I20260928 16:37:04.857137 10708 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:04.857203 10708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:04.861104 10708 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:37:04.863296 10758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:04.864568 10708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:37:04.864718 10708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "c3077d0b67944e20b4338dd1fe0374e4"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "44d1c39437ece18cead9aef2377c648f"
server_key_iv: "2f663d4fc49246537c105428dbc8cb6a"
server_key_version: "encryptionkey@0"
I20260928 16:37:04.865320 10708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:04.878589 10708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:04.881779 10708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:04.881979 10708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:04.882558 10708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:04.883613 10765 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:37:04.888886 10708 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:37:04.888995 10708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 16:37:04.889070 10708 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:37:04.890645 10708 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:37:04.890713 10708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 16:37:04.890990 10765 tablet_bootstrap.cc:492] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4: Bootstrap starting.
I20260928 16:37:04.901517 10708 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:38281
I20260928 16:37:04.901604 10874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:38281 every 8 connection(s)
I20260928 16:37:04.902431 10708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:37:04.909446 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 10708
I20260928 16:37:04.909904 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:41917
--local_ip_for_outbound_sockets=127.27.122.194
--tserver_master_addrs=127.27.122.254:42599
--webserver_port=40413
--webserver_interface=127.27.122.194
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:37:04.910974 10765 log.cc:824] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:04.917907 10765 tablet_bootstrap.cc:492] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:37:04.918262 10765 tablet_bootstrap.cc:492] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4: Bootstrap complete.
I20260928 16:37:04.919509 10765 ts_tablet_manager.cc:1397] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.012s
I20260928 16:37:04.921819 10765 raft_consensus.cc:348] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:04.922235 10765 raft_consensus.cc:729] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3077d0b67944e20b4338dd1fe0374e4, State: Initialized, Role: FOLLOWER
I20260928 16:37:04.922705 10765 consensus_queue.cc:260] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:04.923591 10765 ts_tablet_manager.cc:1428] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
W20260928 16:37:05.013132 10877 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:37:04.904272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:52129)
0928 16:37:04.904785 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:04.904853 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:04.906003 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:05.012302 (+106299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":111,"threads_started":1}
W20260928 16:37:05.013772 10875 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:42599 (0 consecutive failures): Network error: Failed to ping master at 127.27.122.254:42599: Client connection negotiation failed: client connection to 127.27.122.254:42599: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 16:37:05.014570 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:05.014285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:41987)
0928 16:37:05.014417 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:05.014517 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 16:37:05.015316 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:05.015095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:50667)
0928 16:37:05.015209 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:05.015256 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 16:37:05.015524 10875 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 16:37:05.049909 10878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:05.050129 10878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:05.050206 10878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:05.054176 10878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:05.054227 10878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:05.054325 10878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:37:05.059441 10878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:41917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=40413
--tserver_master_addrs=127.27.122.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:05.060410 10878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:05.061494 10878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:05.071002 10886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:05.071066 10889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:05.071128 10878 server_base.cc:1034] running on GCE node
W20260928 16:37:05.071079 10887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:05.071673 10878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:05.072269 10878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:05.073452 10878 hybrid_clock.cc:648] HybridClock initialized: now 1790613425073449 us; error 37 us; skew 500 ppm
Sep 28 16:37:05 dist-test-slave-g9sh krb5kdc[9529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613425, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:05.076148 10878 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:37:05.077116 10878 webserver.cc:460] Webserver started at http://127.27.122.194:40413/ using document root <none> and password file <none>
I20260928 16:37:05.077615 10878 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:05.077680 10878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:05.081363 10878 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 16:37:05.083313 10897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:05.084534 10878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:37:05.084630 10878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "409bc21e24964f16afb28d4abfe22b83"
format_stamp: "Formatted at 2026-09-28 16:37:02 on dist-test-slave-g9sh"
server_key: "52bcf1a903a373db0e8eb58e7eb501e2"
server_key_iv: "a941009f791633fe2984fd6ae4c94027"
server_key_version: "encryptionkey@0"
I20260928 16:37:05.085247 10878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:05.095078 10878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:05.098214 10878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:05.098412 10878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:05.098973 10878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:05.099771 10878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:37:05.099817 10878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:05.099856 10878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:37:05.099911 10878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:05.109489 10878 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:41917
I20260928 16:37:05.109530 11012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:41917 every 8 connection(s)
I20260928 16:37:05.110435 10878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:37:05.114038 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:05.112396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:59941)
0928 16:37:05.112840 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:05.113358 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":107,"threads_started":1}
W20260928 16:37:05.114562 11013 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:42599 (0 consecutive failures): Network error: Failed to ping master at 127.27.122.254:42599: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
I20260928 16:37:05.115068 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:05.114864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:35755)
0928 16:37:05.114965 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:05.115017 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 16:37:05.115698 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:05.115475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:49597)
0928 16:37:05.115600 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:05.115651 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 16:37:05.115866 11013 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:37:05.117412 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 10878
I20260928 16:37:05.117832 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:37303
--local_ip_for_outbound_sockets=127.27.122.195
--tserver_master_addrs=127.27.122.254:42599
--webserver_port=39753
--webserver_interface=127.27.122.195
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:39999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:37:05.229564 11017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:05.229856 11017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:05.229928 11017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:05.234059 11017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:05.234139 11017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:05.234227 11017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:37:05.239599 11017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:39999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:37303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=39753
--tserver_master_addrs=127.27.122.254:42599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:05.240713 11017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:05.241813 11017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:05.251541 11022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:05.251556 11025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:05.251658 11017 server_base.cc:1034] running on GCE node
W20260928 16:37:05.251806 11023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:05.252215 11017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:05.252815 11017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:05.253990 11017 hybrid_clock.cc:648] HybridClock initialized: now 1790613425253936 us; error 63 us; skew 500 ppm
Sep 28 16:37:05 dist-test-slave-g9sh krb5kdc[9529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613425, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:05.256603 11017 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:37:05.257546 11017 webserver.cc:460] Webserver started at http://127.27.122.195:39753/ using document root <none> and password file <none>
I20260928 16:37:05.258098 11017 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:05.258159 11017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:05.261958 11017 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 16:37:05.263958 11032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:05.265192 11017 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:37:05.265308 11017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "9ee50be9d9b040a3b1f11c8af268a660"
format_stamp: "Formatted at 2026-09-28 16:37:03 on dist-test-slave-g9sh"
server_key: "19d28a05aabf8420739334073e831024"
server_key_iv: "dea52b4e47019abcb5d8f8de2379e448"
server_key_version: "encryptionkey@0"
I20260928 16:37:05.265825 11017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:05.285645 11017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:05.288360 11017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:05.288695 11017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:05.289430 11017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:05.290225 11017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:37:05.290272 11017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:05.290341 11017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:37:05.290380 11017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:05.298990 11017 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:37303
I20260928 16:37:05.299037 11147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:37303 every 8 connection(s)
I20260928 16:37:05.299939 11017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:37:05.303438 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:05.301933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:48221)
0928 16:37:05.302369 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:05.302785 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":108,"threads_started":1}
W20260928 16:37:05.303936 11148 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:42599 (0 consecutive failures): Network error: Failed to ping master at 127.27.122.254:42599: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
I20260928 16:37:05.304143 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 11017
I20260928 16:37:05.304581 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:05.304337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:40541)
0928 16:37:05.304443 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:05.304500 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 16:37:05.305297 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:05.305097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:49191)
0928 16:37:05.305200 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:05.305253 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 16:37:05.305451 11148 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:37:06.016559 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:06.016106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:37417)
0928 16:37:06.016385 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:06.016440 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260928 16:37:06.116760 11154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:06.116355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:45497)
0928 16:37:06.116601 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:06.116657 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":85,"threads_started":1}
I20260928 16:37:06.306526 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:06.306170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:44059)
0928 16:37:06.306389 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:06.306437 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260928 16:37:06.569844 11156 raft_consensus.cc:482] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:37:06.570035 11156 raft_consensus.cc:504] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:06.571321 11156 leader_election.cc:288] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303), 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917)
Sep 28 16:37:06 dist-test-slave-g9sh krb5kdc[9529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613424, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.194@KRBTEST.COM
Sep 28 16:37:06 dist-test-slave-g9sh krb5kdc[9529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613424, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.195@KRBTEST.COM
I20260928 16:37:06.589947 11158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:06.571729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:49177 (local address 127.27.122.195:37303)
0928 16:37:06.572185 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:06.572211 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:06.572256 (+ 45us) server_negotiation.cc:407] Connection header received
0928 16:37:06.573034 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:06.573041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:06.573377 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:06.573685 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:06.574949 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:06.575909 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:06.577282 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:06.577738 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:06.577925 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:06.585242 (+ 7317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:06.585248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:06.585265 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:06.585304 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:06.587208 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:06.588011 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:06.588018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:06.588023 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:06.588171 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:06.588480 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:06.588486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:06.588490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:06.588960 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:06.589097 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:06.589482 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 16:37:06.589759 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":124,"threads_started":1}
I20260928 16:37:06.589943 11159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:06.571534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:37303 (local address 127.27.122.193:49177)
0928 16:37:06.572119 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:06.572134 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:06.572261 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:06.573730 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:06.573736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:06.574299 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:06.574759 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:06.574773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:06.576057 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:06.576061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:06.577159 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:06.577169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:06.577864 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:06.577868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:06.578115 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:06.582666 (+ 4551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:06.582693 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:06.585036 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:06.587727 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:06.587729 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:06.587730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:06.587887 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:06.588273 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:06.588277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:06.588279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:06.588363 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:06.589066 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:06.589069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:06.589176 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 16:37:06.589351 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 16:37:06.589735 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":417,"spinlock_wait_cycles":18176,"thread_start_us":86,"threads_started":1}
I20260928 16:37:06.589948 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:06.571806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:45127 (local address 127.27.122.194:41917)
0928 16:37:06.572210 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:06.572236 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:06.572277 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:37:06.573039 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:06.573046 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:06.573376 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:06.573685 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:06.574935 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:06.575923 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:06.577269 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:06.577739 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:06.577924 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:06.584885 (+ 6961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:06.584890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:06.584946 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:06.584990 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:06.587208 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:06.587737 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:06.587741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:06.587746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:06.587861 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:06.588265 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:06.588268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:06.588270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:06.588771 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:06.588944 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:06.589459 (+ 515us) server_negotiation.cc:299] Negotiation successful
0928 16:37:06.589750 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":65,"threads_started":1}
I20260928 16:37:06.590008 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:06.571534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:41917 (local address 127.27.122.193:45127)
0928 16:37:06.572069 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:06.572085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:06.572194 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:06.573730 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:06.573736 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:06.574298 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:06.574758 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:06.574773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:06.576062 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:06.576065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:06.577151 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:06.577161 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:06.577867 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:06.577871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:06.578111 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:06.582666 (+ 4555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:06.582698 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:06.584631 (+ 1933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:06.587334 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:06.587338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:06.587350 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:06.587633 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:06.587969 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:06.587972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:06.587974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:06.588151 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:06.588891 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:06.588914 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:06.589154 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 16:37:06.589351 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 16:37:06.589822 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":383,"spinlock_wait_cycles":33792,"thread_start_us":76,"threads_started":1}
W20260928 16:37:06.593117 10759 leader_election.cc:341] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303): Not found: Tablet not found: c8e3d761036b484286202fe4d56681b8
W20260928 16:37:06.593532 10761 leader_election.cc:341] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917): Not found: Tablet not found: c8e3d761036b484286202fe4d56681b8
I20260928 16:37:06.593606 10761 leader_election.cc:302] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3077d0b67944e20b4338dd1fe0374e4; no voters: 409bc21e24964f16afb28d4abfe22b83, 9ee50be9d9b040a3b1f11c8af268a660
I20260928 16:37:06.593899 11156 raft_consensus.cc:2726] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:37:07.017644 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:07.017371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:54513)
0928 16:37:07.017501 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:07.017555 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:37:07.117795 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:07.117439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:60655)
0928 16:37:07.117663 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:07.117714 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
I20260928 16:37:07.307590 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:07.307214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:54199)
0928 16:37:07.307458 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:07.307509 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
I20260928 16:37:08.018781 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:08.018377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:36621)
0928 16:37:08.018619 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:08.018670 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":85,"threads_started":1}
I20260928 16:37:08.118916 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:08.118530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:36461)
0928 16:37:08.118775 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:08.118828 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260928 16:37:08.308802 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:08.308416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:51615)
0928 16:37:08.308668 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:08.308721 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260928 16:37:08.581884 11166 raft_consensus.cc:482] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:37:08.582015 11166 raft_consensus.cc:504] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:08.582472 11166 leader_election.cc:288] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303), 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917)
W20260928 16:37:08.583364 10761 leader_election.cc:341] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917): Not found: Tablet not found: c8e3d761036b484286202fe4d56681b8
W20260928 16:37:08.583452 10759 leader_election.cc:341] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303): Not found: Tablet not found: c8e3d761036b484286202fe4d56681b8
I20260928 16:37:08.583570 10759 leader_election.cc:302] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3077d0b67944e20b4338dd1fe0374e4; no voters: 409bc21e24964f16afb28d4abfe22b83, 9ee50be9d9b040a3b1f11c8af268a660
I20260928 16:37:08.583729 11166 raft_consensus.cc:2726] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:37:09.020035 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:09.019615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:52485)
0928 16:37:09.019859 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:09.019923 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":87,"threads_started":1}
I20260928 16:37:09.119925 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:09.119545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:48573)
0928 16:37:09.119789 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:09.119842 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":74,"threads_started":1}
I20260928 16:37:09.309943 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:09.309589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:38175)
0928 16:37:09.309807 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:09.309858 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260928 16:37:10.021593 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:10.021135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:53599)
0928 16:37:10.021423 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:10.021496 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":72,"threads_started":1}
I20260928 16:37:10.121200 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:10.120746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:42445)
0928 16:37:10.121057 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:10.121110 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20260928 16:37:10.222877 11172 raft_consensus.cc:482] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:37:10.223044 11172 raft_consensus.cc:504] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:10.223536 11172 leader_election.cc:288] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303), 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917)
W20260928 16:37:10.224310 10761 leader_election.cc:341] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917): Not found: Tablet not found: c8e3d761036b484286202fe4d56681b8
W20260928 16:37:10.224408 10759 leader_election.cc:341] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303): Not found: Tablet not found: c8e3d761036b484286202fe4d56681b8
I20260928 16:37:10.224479 10759 leader_election.cc:302] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3077d0b67944e20b4338dd1fe0374e4; no voters: 409bc21e24964f16afb28d4abfe22b83, 9ee50be9d9b040a3b1f11c8af268a660
I20260928 16:37:10.224656 11172 raft_consensus.cc:2726] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:37:10.311110 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:10.310705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:45199)
0928 16:37:10.310965 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:10.311023 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
I20260928 16:37:11.022765 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:11.022362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:36109)
0928 16:37:11.022614 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:11.022667 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260928 16:37:11.122229 11175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:11.121849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:48105)
0928 16:37:11.122091 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:11.122144 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":72,"threads_started":1}
I20260928 16:37:11.312160 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:11.311776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:53297)
0928 16:37:11.312027 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:11.312078 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":91,"threads_started":1}
I20260928 16:37:12.023902 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:12.023495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:56273)
0928 16:37:12.023753 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:12.023805 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20260928 16:37:12.123463 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:12.123028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:40795)
0928 16:37:12.123322 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:12.123378 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":79,"threads_started":1}
I20260928 16:37:12.313441 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:12.312991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:55869)
0928 16:37:12.313303 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:12.313357 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":81,"threads_started":1}
I20260928 16:37:13.025074 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:13.024596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:58075)
0928 16:37:13.024866 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:13.024973 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":97,"threads_started":1}
I20260928 16:37:13.124732 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:13.124346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:59881)
0928 16:37:13.124586 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:13.124644 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260928 16:37:13.314689 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:13.314270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:58367)
0928 16:37:13.314549 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:13.314601 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":119,"threads_started":1}
I20260928 16:37:13.887053 11183 raft_consensus.cc:482] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:37:13.887207 11183 raft_consensus.cc:504] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:13.887653 11183 leader_election.cc:288] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303), 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917)
W20260928 16:37:13.888365 10759 leader_election.cc:341] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303): Not found: Tablet not found: c8e3d761036b484286202fe4d56681b8
W20260928 16:37:13.888511 10761 leader_election.cc:341] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917): Not found: Tablet not found: c8e3d761036b484286202fe4d56681b8
I20260928 16:37:13.888612 10761 leader_election.cc:302] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3077d0b67944e20b4338dd1fe0374e4; no voters: 409bc21e24964f16afb28d4abfe22b83, 9ee50be9d9b040a3b1f11c8af268a660
I20260928 16:37:13.888799 11183 raft_consensus.cc:2726] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:37:14.026218 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:14.025791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:34479)
0928 16:37:14.026049 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:14.026105 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260928 16:37:14.125895 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:14.125508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:38159)
0928 16:37:14.125754 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:14.125809 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260928 16:37:14.315838 11186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:14.315418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:59161)
0928 16:37:14.315677 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:14.315744 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260928 16:37:15.027486 11187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:15.027076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:56899)
0928 16:37:15.027331 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:15.027386 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260928 16:37:15.127027 11188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:15.126598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:49835)
0928 16:37:15.126872 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:15.126932 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":108,"threads_started":1}
I20260928 16:37:15.317065 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:15.316650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:54567)
0928 16:37:15.316872 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:15.316979 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
I20260928 16:37:16.028784 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:16.028333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:53983)
0928 16:37:16.028607 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:16.028666 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
I20260928 16:37:16.128250 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:16.127741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:59929)
0928 16:37:16.128075 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:16.128135 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":125,"threads_started":1}
I20260928 16:37:16.318172 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:16.317759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:39253)
0928 16:37:16.318015 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:16.318071 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260928 16:37:17.029991 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:17.029582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:40411)
0928 16:37:17.029839 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:17.029891 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260928 16:37:17.129547 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:17.129115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:57575)
0928 16:37:17.129365 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:17.129421 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":77,"threads_started":1}
I20260928 16:37:17.319279 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:17.318871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:36599)
0928 16:37:17.319120 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:17.319187 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260928 16:37:17.590276 11196 raft_consensus.cc:482] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:37:17.590408 11196 raft_consensus.cc:504] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ee50be9d9b040a3b1f11c8af268a660" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37303 } } peers { permanent_uuid: "409bc21e24964f16afb28d4abfe22b83" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41917 } } peers { permanent_uuid: "c3077d0b67944e20b4338dd1fe0374e4" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 38281 } }
I20260928 16:37:17.590902 11196 leader_election.cc:288] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303), 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917)
W20260928 16:37:17.591681 10759 leader_election.cc:341] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ee50be9d9b040a3b1f11c8af268a660 (127.27.122.195:37303): Not found: Tablet not found: c8e3d761036b484286202fe4d56681b8
W20260928 16:37:17.591835 10761 leader_election.cc:341] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 409bc21e24964f16afb28d4abfe22b83 (127.27.122.194:41917): Not found: Tablet not found: c8e3d761036b484286202fe4d56681b8
I20260928 16:37:17.591921 10761 leader_election.cc:302] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3077d0b67944e20b4338dd1fe0374e4; no voters: 409bc21e24964f16afb28d4abfe22b83, 9ee50be9d9b040a3b1f11c8af268a660
I20260928 16:37:17.592099 11196 raft_consensus.cc:2726] T c8e3d761036b484286202fe4d56681b8 P c3077d0b67944e20b4338dd1fe0374e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 16:37:18.031211 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:18.030829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:54923)
0928 16:37:18.031065 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:18.031115 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260928 16:37:18.130728 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:18.130294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:36625)
0928 16:37:18.130576 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:18.130632 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":85,"threads_started":1}
I20260928 16:37:18.320394 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:18.319969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:39787)
0928 16:37:18.320243 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:18.320301 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":70,"threads_started":1}
I20260928 16:37:19.032371 11200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:19.031960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:34295)
0928 16:37:19.032217 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:19.032271 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":77,"threads_started":1}
I20260928 16:37:19.132076 11201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:19.131615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:60317)
0928 16:37:19.131919 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:19.131971 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":93,"threads_started":1}
I20260928 16:37:19.321601 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:19.321203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.195:52885)
0928 16:37:19.321449 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:19.321507 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":82,"threads_started":1}
I20260928 16:37:20.033721 11203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:20.033305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.193:58729)
0928 16:37:20.033566 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:20.033622 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":76,"threads_started":1}
I20260928 16:37:20.133354 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:20.132882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:42599 (local address 127.27.122.194:43451)
0928 16:37:20.133206 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:20.133259 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:42599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":80,"threads_started":1}
I20260928 16:37:20.308517 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 10708
I20260928 16:37:20.314860 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 10878
I20260928 16:37:20.320648 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 11017
2026-09-28T16:37:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19271 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:37:20 dist-test-slave-g9sh krb5kdc[11207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:37:20 dist-test-slave-g9sh krb5kdc[11207](info): set up 2 sockets
Sep 28 16:37:20 dist-test-slave-g9sh krb5kdc[11207](info): commencing operation
krb5kdc: starting...
W20260928 16:37:21.371793 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:37:21 dist-test-slave-g9sh krb5kdc[11207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:37:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:37:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:37:21.492393 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:38467
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:38467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:37:21.600219 11221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:21.600463 11221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:21.600502 11221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:21.604307 11221 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:37:21.604347 11221 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:37:21.604363 11221 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:37:21.604377 11221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:37:21.604394 11221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:21.604411 11221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:21.604432 11221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:37:21.604447 11221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:37:21.609447 11221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:38467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:38467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:21.610283 11221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:21.611153 11221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:21.619378 11226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:21.619443 11229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:21.619495 11227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:21.619557 11221 server_base.cc:1034] running on GCE node
I20260928 16:37:21.620289 11221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:21.621323 11221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:21.622545 11221 hybrid_clock.cc:648] HybridClock initialized: now 1790613441622512 us; error 42 us; skew 500 ppm
Sep 28 16:37:21 dist-test-slave-g9sh krb5kdc[11207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613441, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:21.625213 11221 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:37:21.626084 11221 webserver.cc:460] Webserver started at http://127.27.122.254:43401/ using document root <none> and password file <none>
I20260928 16:37:21.626603 11221 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:21.626662 11221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:21.626889 11221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:37:21.628790 11221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "ccbef86f21314323a24936b454aa1ce4"
format_stamp: "Formatted at 2026-09-28 16:37:21 on dist-test-slave-g9sh"
server_key: "6a644e0d6760e1937867266f5ae50600"
server_key_iv: "945ba9fbd8628bfc062a4c25aeaaa81d"
server_key_version: "encryptionkey@0"
I20260928 16:37:21.629405 11221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "ccbef86f21314323a24936b454aa1ce4"
format_stamp: "Formatted at 2026-09-28 16:37:21 on dist-test-slave-g9sh"
server_key: "6a644e0d6760e1937867266f5ae50600"
server_key_iv: "945ba9fbd8628bfc062a4c25aeaaa81d"
server_key_version: "encryptionkey@0"
I20260928 16:37:21.632989 11221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:37:21.635238 11236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:21.636394 11221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:37:21.636538 11221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "ccbef86f21314323a24936b454aa1ce4"
format_stamp: "Formatted at 2026-09-28 16:37:21 on dist-test-slave-g9sh"
server_key: "6a644e0d6760e1937867266f5ae50600"
server_key_iv: "945ba9fbd8628bfc062a4c25aeaaa81d"
server_key_version: "encryptionkey@0"
I20260928 16:37:21.636644 11221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:21.661908 11221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:21.664661 11221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:21.664860 11221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:21.671854 11221 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:38467
I20260928 16:37:21.672138 11289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:38467 every 8 connection(s)
I20260928 16:37:21.672793 11221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:37:21.675539 11290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:21.677989 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 11221
I20260928 16:37:21.678157 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:37:21.678459 28139 external_mini_cluster.cc:1428] Setting key 404e64274d4acbb9524d0c4570cf2c2a
I20260928 16:37:21.681542 11290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Bootstrap starting.
I20260928 16:37:21.683699 11290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:21.684412 11290 log.cc:824] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:21.686383 11290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: No bootstrap required, opened a new log
Sep 28 16:37:21 dist-test-slave-g9sh krb5kdc[11207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:37:21.689797 11290 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } }
I20260928 16:37:21.690006 11290 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:21.690038 11290 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbef86f21314323a24936b454aa1ce4, State: Initialized, Role: FOLLOWER
I20260928 16:37:21.690531 11290 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } }
I20260928 16:37:21.690682 11290 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:37:21.690747 11290 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:37:21.690845 11290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:21.691679 11290 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } }
I20260928 16:37:21.692013 11290 leader_election.cc:302] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccbef86f21314323a24936b454aa1ce4; no voters:
I20260928 16:37:21.692337 11290 leader_election.cc:288] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:37:21.692397 11295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:37:21.692701 11295 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 1 LEADER]: Becoming Leader. State: Replica: ccbef86f21314323a24936b454aa1ce4, State: Running, Role: LEADER
I20260928 16:37:21.693068 11295 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } }
I20260928 16:37:21.693642 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:21.679868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.27.122.254:38467)
0928 16:37:21.680283 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:21.680297 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:21.680331 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:37:21.681058 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:21.681077 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:21.681401 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:21.681743 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:21.682548 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:21.683510 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:21.684448 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:21.685025 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:21.685196 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:21.687924 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:21.687930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:21.687942 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:21.687968 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:21.690451 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:21.691012 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:21.691018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:21.691023 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:21.691130 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:21.691455 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:21.691458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:21.691459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:21.691874 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:21.692070 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:21.692345 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 16:37:21.692692 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":115,"threads_started":1}
I20260928 16:37:21.693702 11290 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:37:21.694720 11297 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccbef86f21314323a24936b454aa1ce4. Latest consensus state: current_term: 1 leader_uuid: "ccbef86f21314323a24936b454aa1ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } } }
I20260928 16:37:21.694857 11297 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:37:21.695199 11296 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccbef86f21314323a24936b454aa1ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } } }
I20260928 16:37:21.695256 11304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:37:21.695286 11296 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:37:21.698295 11304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:37:21.704015 11304 catalog_manager.cc:1347] Generated new cluster ID: 9fddab6d07004348b753b5e417948ea2
I20260928 16:37:21.704077 11304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:37:21.711962 11304 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:37:21.712810 11304 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:37:21.722216 11304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Generated new TSK 0
I20260928 16:37:21.722801 11304 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:37:21.762295 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:38467
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:37:21.873075 11316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:21.873359 11316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:21.873440 11316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:21.877385 11316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:21.877475 11316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:21.877563 11316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:37:21.882663 11316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:21.883708 11316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:21.884742 11316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:37:21.894475 11316 server_base.cc:1034] running on GCE node
W20260928 16:37:21.894469 11322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:21.894464 11325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:21.894624 11323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:21.895169 11316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:21.895818 11316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:21.896996 11316 hybrid_clock.cc:648] HybridClock initialized: now 1790613441896972 us; error 34 us; skew 500 ppm
Sep 28 16:37:21 dist-test-slave-g9sh krb5kdc[11207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613441, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:21.899704 11316 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:37:21.900859 11316 webserver.cc:460] Webserver started at http://127.27.122.193:38881/ using document root <none> and password file <none>
I20260928 16:37:21.901520 11316 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:21.901583 11316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:21.901741 11316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:37:21.903669 11316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "1461e8d6cf9b4e98bd773aff186b5a01"
format_stamp: "Formatted at 2026-09-28 16:37:21 on dist-test-slave-g9sh"
server_key: "2622e725b6be020739ab2c608273574a"
server_key_iv: "bddd22a38adc7af75a31e31024d4f6c9"
server_key_version: "encryptionkey@0"
I20260928 16:37:21.904162 11316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "1461e8d6cf9b4e98bd773aff186b5a01"
format_stamp: "Formatted at 2026-09-28 16:37:21 on dist-test-slave-g9sh"
server_key: "2622e725b6be020739ab2c608273574a"
server_key_iv: "bddd22a38adc7af75a31e31024d4f6c9"
server_key_version: "encryptionkey@0"
I20260928 16:37:21.907714 11316 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:37:21.910171 11332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:21.911478 11316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:37:21.911639 11316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "1461e8d6cf9b4e98bd773aff186b5a01"
format_stamp: "Formatted at 2026-09-28 16:37:21 on dist-test-slave-g9sh"
server_key: "2622e725b6be020739ab2c608273574a"
server_key_iv: "bddd22a38adc7af75a31e31024d4f6c9"
server_key_version: "encryptionkey@0"
I20260928 16:37:21.911760 11316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:21.938072 11316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:21.941408 11316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:21.941625 11316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:21.942248 11316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:21.943109 11316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:37:21.943184 11316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:21.943259 11316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:37:21.943301 11316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:21.952340 11316 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:37977
I20260928 16:37:21.952363 11449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:37977 every 8 connection(s)
I20260928 16:37:21.953233 11316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:37:21.958964 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 11316
I20260928 16:37:21.959144 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:37:21.959435 28139 external_mini_cluster.cc:1428] Setting key 0c08cd0f9c94282d1381064aa8597d60
Sep 28 16:37:21 dist-test-slave-g9sh krb5kdc[11207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613441, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:37:21.967936 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:21.955292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:51395 (local address 127.27.122.254:38467)
0928 16:37:21.955449 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:21.955453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:21.956314 (+ 861us) server_negotiation.cc:407] Connection header received
0928 16:37:21.957674 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:21.957677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:21.957725 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:21.957801 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:21.958975 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:21.959689 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:21.961121 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:21.961367 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:21.961504 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:21.964199 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:21.964203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:21.964206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:21.964238 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:21.965811 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:21.966352 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:21.966356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:21.966358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:21.966446 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:21.966817 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:21.966820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:21.966823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:21.967016 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:21.967127 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:21.967632 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 16:37:21.967763 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:37:21.968787 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:21.955687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:51395)
0928 16:37:21.956154 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:21.956197 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:21.957466 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:21.957968 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:21.957974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:21.958317 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:21.958809 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:21.958822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:21.959816 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:21.959820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:21.961001 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:21.961011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:21.961541 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:21.961546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:21.961720 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:21.962323 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:21.962340 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:21.964024 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:21.965945 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:21.965958 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:21.965974 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:21.966240 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:21.966550 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:21.966554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:21.966556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:21.966710 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:21.967128 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:21.967136 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:21.967385 (+ 249us) client_negotiation.cc:769] Sending connection context
0928 16:37:21.967580 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 16:37:21.967903 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":119,"threads_started":1}
I20260928 16:37:21.970078 11450 heartbeater.cc:347] Connected to a master server at 127.27.122.254:38467
I20260928 16:37:21.970328 11450 heartbeater.cc:464] Registering TS with master...
I20260928 16:37:21.970927 11450 heartbeater.cc:511] Master 127.27.122.254:38467 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
I20260928 16:37:21.972595 11253 ts_manager.cc:194] Registered new tserver with Master: 1461e8d6cf9b4e98bd773aff186b5a01 (127.27.122.193:37977)
I20260928 16:37:21.974133 11253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:51395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:37:21.987735 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:38467
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:37:22.097317 11455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:22.097548 11455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:22.097611 11455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:22.101622 11455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:22.101673 11455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:22.101761 11455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:37:22.106823 11455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:22.107887 11455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:22.108954 11455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:22.118896 11463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:22.118896 11461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:22.119031 11460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:22.119228 11455 server_base.cc:1034] running on GCE node
I20260928 16:37:22.119638 11455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:22.120268 11455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:22.121466 11455 hybrid_clock.cc:648] HybridClock initialized: now 1790613442121436 us; error 46 us; skew 500 ppm
Sep 28 16:37:22 dist-test-slave-g9sh krb5kdc[11207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613442, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:22.124173 11455 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:37:22.125027 11455 webserver.cc:460] Webserver started at http://127.27.122.194:42437/ using document root <none> and password file <none>
I20260928 16:37:22.125595 11455 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:22.125658 11455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:22.125871 11455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:37:22.127869 11455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7"
format_stamp: "Formatted at 2026-09-28 16:37:22 on dist-test-slave-g9sh"
server_key: "cc056c90f6aa34eb8b5d026b4dbbbc7b"
server_key_iv: "98d08574fae11adc5af1b23593cc5867"
server_key_version: "encryptionkey@0"
I20260928 16:37:22.128417 11455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7"
format_stamp: "Formatted at 2026-09-28 16:37:22 on dist-test-slave-g9sh"
server_key: "cc056c90f6aa34eb8b5d026b4dbbbc7b"
server_key_iv: "98d08574fae11adc5af1b23593cc5867"
server_key_version: "encryptionkey@0"
I20260928 16:37:22.132035 11455 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 16:37:22.134565 11470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:22.135762 11455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:37:22.135857 11455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7"
format_stamp: "Formatted at 2026-09-28 16:37:22 on dist-test-slave-g9sh"
server_key: "cc056c90f6aa34eb8b5d026b4dbbbc7b"
server_key_iv: "98d08574fae11adc5af1b23593cc5867"
server_key_version: "encryptionkey@0"
I20260928 16:37:22.135988 11455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:22.150842 11455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:22.153692 11455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:22.153888 11455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:22.154469 11455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:22.155346 11455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:37:22.155393 11455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:22.155457 11455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:37:22.155493 11455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:22.164294 11455 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:41341
I20260928 16:37:22.164323 11586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:41341 every 8 connection(s)
I20260928 16:37:22.165191 11455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:37:22.173570 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 11455
I20260928 16:37:22.173729 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:37:22.174036 28139 external_mini_cluster.cc:1428] Setting key e62f46badc801ec1a177284167919651
Sep 28 16:37:22 dist-test-slave-g9sh krb5kdc[11207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613442, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:37:22.179585 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.167187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:46533 (local address 127.27.122.254:38467)
0928 16:37:22.167317 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:22.167322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:22.168236 (+ 914us) server_negotiation.cc:407] Connection header received
0928 16:37:22.169609 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:22.169612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:22.169659 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:22.169735 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:22.170927 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.171463 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.172673 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.172892 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.173041 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.175603 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:22.175606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:22.175608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:22.175630 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:22.177197 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:22.177834 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:22.177838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:22.177839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:22.177920 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:22.178335 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:22.178341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:22.178346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:22.178596 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:22.178706 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:22.179211 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 16:37:22.179351 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 16:37:22.180199 11589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.167617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:46533)
0928 16:37:22.168073 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:22.168121 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:22.169376 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:22.169920 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:22.169927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:22.170287 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:22.170791 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.170805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.171606 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.171609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.172550 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.172560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.173084 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.173087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.173229 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.173907 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:22.173923 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:22.175446 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:22.177312 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:22.177321 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:22.177336 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:22.177702 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:22.178026 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:22.178031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:22.178033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:22.178223 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:22.178704 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:22.178708 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:22.178943 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 16:37:22.179098 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 16:37:22.179351 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":103,"threads_started":1}
I20260928 16:37:22.181576 11587 heartbeater.cc:347] Connected to a master server at 127.27.122.254:38467
I20260928 16:37:22.181842 11587 heartbeater.cc:464] Registering TS with master...
I20260928 16:37:22.182435 11587 heartbeater.cc:511] Master 127.27.122.254:38467 requested a full tablet report, sending...
I20260928 16:37:22.183660 11253 ts_manager.cc:194] Registered new tserver with Master: f4821abf0a5b4e1c90d7ebcfb83e51d7 (127.27.122.194:41341)
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
I20260928 16:37:22.184325 11253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:46533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:37:22.203456 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:38467
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:37:22.315141 11594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:22.315399 11594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:22.315471 11594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:22.319531 11594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:22.319605 11594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:22.319690 11594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:37:22.324790 11594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:22.325825 11594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:22.326797 11594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:22.336233 11602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:22.336261 11599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:22.336278 11600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:22.336478 11594 server_base.cc:1034] running on GCE node
I20260928 16:37:22.336879 11594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:22.337494 11594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:22.338691 11594 hybrid_clock.cc:648] HybridClock initialized: now 1790613442338677 us; error 24 us; skew 500 ppm
Sep 28 16:37:22 dist-test-slave-g9sh krb5kdc[11207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613442, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:22.341713 11594 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:37:22.342511 11594 webserver.cc:460] Webserver started at http://127.27.122.195:36979/ using document root <none> and password file <none>
I20260928 16:37:22.343068 11594 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:22.343158 11594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:22.343384 11594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:37:22.345422 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "c2ff3fe984f54f04a56ed3377d88fdda"
format_stamp: "Formatted at 2026-09-28 16:37:22 on dist-test-slave-g9sh"
server_key: "c3f15835e4b5f39b13cb9d74cb65dc7f"
server_key_iv: "cd7f9afa8586d2ef2949a2066399fe21"
server_key_version: "encryptionkey@0"
I20260928 16:37:22.345983 11594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "c2ff3fe984f54f04a56ed3377d88fdda"
format_stamp: "Formatted at 2026-09-28 16:37:22 on dist-test-slave-g9sh"
server_key: "c3f15835e4b5f39b13cb9d74cb65dc7f"
server_key_iv: "cd7f9afa8586d2ef2949a2066399fe21"
server_key_version: "encryptionkey@0"
I20260928 16:37:22.349495 11594 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 16:37:22.351755 11609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:22.353020 11594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:37:22.353140 11594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "c2ff3fe984f54f04a56ed3377d88fdda"
format_stamp: "Formatted at 2026-09-28 16:37:22 on dist-test-slave-g9sh"
server_key: "c3f15835e4b5f39b13cb9d74cb65dc7f"
server_key_iv: "cd7f9afa8586d2ef2949a2066399fe21"
server_key_version: "encryptionkey@0"
I20260928 16:37:22.353251 11594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:22.375368 11594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:22.378500 11594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:22.378690 11594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:22.379271 11594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:22.380066 11594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:37:22.380134 11594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:22.380204 11594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:37:22.380246 11594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:22.389043 11594 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:35433
I20260928 16:37:22.389086 11723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:35433 every 8 connection(s)
I20260928 16:37:22.389900 11594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
Sep 28 16:37:22 dist-test-slave-g9sh krb5kdc[11207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613442, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:37:22.399827 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 11594
I20260928 16:37:22.399969 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:37:22.400267 28139 external_mini_cluster.cc:1428] Setting key e9db721fce9fd9b139e1b75ee14ff655
I20260928 16:37:22.403216 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.391630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:36403 (local address 127.27.122.254:38467)
0928 16:37:22.391786 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:22.391791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:22.392631 (+ 840us) server_negotiation.cc:407] Connection header received
0928 16:37:22.393943 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:22.393946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:22.394010 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:22.394083 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:22.395346 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.395935 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.397053 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.397262 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.397340 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.399564 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:22.399567 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:22.399569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:22.399592 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:22.401250 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:22.401786 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:22.401793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:22.401795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:22.401880 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:22.402251 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:22.402256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:22.402258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:22.402484 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:22.402587 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:22.402995 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 16:37:22.403093 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:37:22.404029 11726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.392051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:36403)
0928 16:37:22.392471 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:22.392516 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:22.393731 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:22.394241 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:22.394247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:22.394691 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:22.395221 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.395236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.396049 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.396052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.396944 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.396963 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.397386 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.397388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.397531 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.398046 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:22.398061 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:22.399420 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:22.401357 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:22.401362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:22.401375 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:22.401706 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:22.401977 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:22.401980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:22.401982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:22.402122 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:22.402587 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:22.402591 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:22.402800 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 16:37:22.402946 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 16:37:22.403178 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":102,"threads_started":1}
I20260928 16:37:22.405202 11724 heartbeater.cc:347] Connected to a master server at 127.27.122.254:38467
I20260928 16:37:22.405438 11724 heartbeater.cc:464] Registering TS with master...
I20260928 16:37:22.405872 11724 heartbeater.cc:511] Master 127.27.122.254:38467 requested a full tablet report, sending...
I20260928 16:37:22.406803 11253 ts_manager.cc:194] Registered new tserver with Master: c2ff3fe984f54f04a56ed3377d88fdda (127.27.122.195:35433)
I20260928 16:37:22.407305 11253 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:36403
I20260928 16:37:22.414073 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:37:22.423205 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.416117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.27.122.254:38467)
0928 16:37:22.416250 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:22.416254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:22.416427 (+ 173us) server_negotiation.cc:407] Connection header received
0928 16:37:22.416580 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:22.416583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:22.416619 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:22.416690 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:22.417479 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.417960 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.418757 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.418954 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.419026 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.420262 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:22.420265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:22.420267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:22.420288 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:22.421807 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:22.422168 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:22.422171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:22.422172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:22.422240 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:22.422459 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:22.422461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:22.422462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:22.422661 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:22.422761 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:22.422927 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 16:37:22.423011 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:37:22.425875 11253 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:37:22.428861 11253 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:37:22.443053 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.436832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:37977 (local address 127.0.0.1:55572)
0928 16:37:22.437207 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:22.437262 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:22.437391 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:22.437996 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:22.438004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:22.438030 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:22.438194 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.438201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.439821 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.439826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.441324 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.441336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.442478 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.442482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.442597 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.442638 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 16:37:22.442770 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 16:37:22.442889 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":124,"threads_started":1}
I20260928 16:37:22.443199 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.437588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:41341 (local address 127.0.0.1:57260)
0928 16:37:22.438116 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:22.438126 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:22.438233 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:22.438855 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:22.438858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:22.438891 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:22.439190 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.439195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.440629 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.440632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.441840 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.441850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.442807 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.442811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.442901 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.442925 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:37:22.442998 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 16:37:22.443066 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":224,"spinlock_wait_cycles":5376,"thread_start_us":92,"threads_started":1}
I20260928 16:37:22.443435 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.437221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:35433 (local address 127.0.0.1:35864)
0928 16:37:22.437831 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:22.437848 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:22.437936 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:22.438855 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:22.438858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:22.438892 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:22.439015 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.439023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.440787 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.440793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.441967 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.441978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.443034 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.443037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.443161 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.443196 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:37:22.443267 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 16:37:22.443335 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":7,"spinlock_wait_cycles":5120,"thread_start_us":106,"threads_started":1}
I20260928 16:37:22.443671 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.437048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.27.122.193:37977)
0928 16:37:22.437421 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:22.437439 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:22.437467 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:37:22.437550 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:22.437556 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:22.437781 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:22.438359 (+ 578us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:22.438551 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.439661 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.441482 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.442351 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.442480 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.442610 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:22.443433 (+ 823us) server_negotiation.cc:299] Negotiation successful
0928 16:37:22.443540 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":82,"threads_started":1}
I20260928 16:37:22.443671 11740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.437648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35864 (local address 127.27.122.195:35433)
0928 16:37:22.438475 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:22.438486 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:22.438505 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:37:22.438569 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:22.438573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:22.438712 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:22.439372 (+ 660us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:22.439541 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.440655 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.442103 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.442906 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.443048 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.443399 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:22.443446 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 16:37:22.443531 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":178,"threads_started":1}
I20260928 16:37:22.444125 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.438007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.27.122.194:41341)
0928 16:37:22.438361 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:22.438378 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:22.438402 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:37:22.438481 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:22.438487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:22.438689 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:22.438959 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:22.439301 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.440483 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.441968 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.442692 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.442805 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.442923 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:22.443869 (+ 946us) server_negotiation.cc:299] Negotiation successful
0928 16:37:22.443981 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":125,"threads_started":1}
I20260928 16:37:22.446435 11657 tablet_service.cc:1467] Processing CreateTablet for tablet 4da6291ac35442b5a41104adacabc757 (DEFAULT_TABLE table=test-table [id=14546e715d114bd4a7c82821dd63f35f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:37:22.446434 11380 tablet_service.cc:1467] Processing CreateTablet for tablet 4da6291ac35442b5a41104adacabc757 (DEFAULT_TABLE table=test-table [id=14546e715d114bd4a7c82821dd63f35f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:37:22.447383 11380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da6291ac35442b5a41104adacabc757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:22.447393 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da6291ac35442b5a41104adacabc757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:22.450644 11521 tablet_service.cc:1467] Processing CreateTablet for tablet 4da6291ac35442b5a41104adacabc757 (DEFAULT_TABLE table=test-table [id=14546e715d114bd4a7c82821dd63f35f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:37:22.451690 11521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da6291ac35442b5a41104adacabc757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:22.454300 11742 tablet_bootstrap.cc:492] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Bootstrap starting.
I20260928 16:37:22.456302 11742 tablet_bootstrap.cc:654] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:22.457512 11742 log.cc:824] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:22.458297 11743 tablet_bootstrap.cc:492] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda: Bootstrap starting.
I20260928 16:37:22.458305 11745 tablet_bootstrap.cc:492] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Bootstrap starting.
I20260928 16:37:22.459681 11742 tablet_bootstrap.cc:492] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: No bootstrap required, opened a new log
I20260928 16:37:22.459900 11742 ts_tablet_manager.cc:1397] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260928 16:37:22.461086 11745 tablet_bootstrap.cc:654] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:22.461086 11743 tablet_bootstrap.cc:654] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:22.461969 11743 log.cc:824] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:22.461978 11745 log.cc:824] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:22.463349 11742 raft_consensus.cc:348] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } }
I20260928 16:37:22.463622 11742 raft_consensus.cc:374] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:22.463728 11742 raft_consensus.cc:729] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1461e8d6cf9b4e98bd773aff186b5a01, State: Initialized, Role: FOLLOWER
I20260928 16:37:22.463807 11743 tablet_bootstrap.cc:492] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda: No bootstrap required, opened a new log
I20260928 16:37:22.463963 11745 tablet_bootstrap.cc:492] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7: No bootstrap required, opened a new log
I20260928 16:37:22.464056 11743 ts_tablet_manager.cc:1397] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20260928 16:37:22.464181 11745 ts_tablet_manager.cc:1397] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260928 16:37:22.464206 11742 consensus_queue.cc:260] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } }
I20260928 16:37:22.464876 11450 heartbeater.cc:503] Master 127.27.122.254:38467 was elected leader, sending a full tablet report...
I20260928 16:37:22.465272 11742 ts_tablet_manager.cc:1428] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:37:22.467690 11745 raft_consensus.cc:348] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } }
I20260928 16:37:22.467993 11745 raft_consensus.cc:374] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:22.468087 11745 raft_consensus.cc:729] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4821abf0a5b4e1c90d7ebcfb83e51d7, State: Initialized, Role: FOLLOWER
I20260928 16:37:22.467947 11743 raft_consensus.cc:348] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } }
I20260928 16:37:22.468227 11743 raft_consensus.cc:374] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:22.468286 11743 raft_consensus.cc:729] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ff3fe984f54f04a56ed3377d88fdda, State: Initialized, Role: FOLLOWER
I20260928 16:37:22.468573 11745 consensus_queue.cc:260] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } }
I20260928 16:37:22.468688 11743 consensus_queue.cc:260] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } }
I20260928 16:37:22.469525 11745 ts_tablet_manager.cc:1428] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:37:22.469671 11743 ts_tablet_manager.cc:1428] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:37:22.469864 11587 heartbeater.cc:503] Master 127.27.122.254:38467 was elected leader, sending a full tablet report...
I20260928 16:37:22.470050 11724 heartbeater.cc:503] Master 127.27.122.254:38467 was elected leader, sending a full tablet report...
I20260928 16:37:22.472380 11748 raft_consensus.cc:482] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:37:22.472534 11748 raft_consensus.cc:504] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } }
I20260928 16:37:22.473656 11748 leader_election.cc:288] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4821abf0a5b4e1c90d7ebcfb83e51d7 (127.27.122.194:41341), c2ff3fe984f54f04a56ed3377d88fdda (127.27.122.195:35433)
I20260928 16:37:22.477681 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.474348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:33609 (local address 127.27.122.194:41341)
0928 16:37:22.474456 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:22.474460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:22.474474 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:37:22.474521 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:22.474524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:22.474576 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:22.474656 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:22.475048 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.475569 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.476621 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.477109 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.477223 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.477274 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:22.477494 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 16:37:22.477564 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:37:22.477696 11740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.474084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:54995 (local address 127.27.122.195:35433)
0928 16:37:22.474214 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:22.474218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:22.474499 (+ 281us) server_negotiation.cc:407] Connection header received
0928 16:37:22.474623 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:22.474626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:22.474673 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:22.474752 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:22.475078 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.475559 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.476665 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.477130 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.477263 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.477315 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:22.477523 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 16:37:22.477593 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:37:22.477705 11752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.473878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:41341 (local address 127.27.122.193:33609)
0928 16:37:22.474199 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:22.474213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:22.474311 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:22.474777 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:22.474780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:22.474825 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:22.474941 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.474946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.475709 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.475712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.476501 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.476510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.477232 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.477235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.477343 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.477366 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:37:22.477443 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:37:22.477533 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":4,"spinlock_wait_cycles":25472,"thread_start_us":71,"threads_started":1}
I20260928 16:37:22.477710 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.473879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:35433 (local address 127.27.122.193:54995)
0928 16:37:22.474396 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:22.474410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:22.474516 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:22.474763 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:22.474766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:22.474782 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:22.474964 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.474971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.475667 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.475670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.476514 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:22.476523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.477269 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.477273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:22.477385 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.477402 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 16:37:22.477497 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:37:22.477564 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":61,"spinlock_wait_cycles":12672,"thread_start_us":82,"threads_started":1}
I20260928 16:37:22.478251 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da6291ac35442b5a41104adacabc757" candidate_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" is_pre_election: true
I20260928 16:37:22.478508 11677 raft_consensus.cc:2454] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1461e8d6cf9b4e98bd773aff186b5a01 in term 0.
I20260928 16:37:22.478657 11541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da6291ac35442b5a41104adacabc757" candidate_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" is_pre_election: true
I20260928 16:37:22.478915 11541 raft_consensus.cc:2454] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1461e8d6cf9b4e98bd773aff186b5a01 in term 0.
I20260928 16:37:22.478979 11336 leader_election.cc:302] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1461e8d6cf9b4e98bd773aff186b5a01, c2ff3fe984f54f04a56ed3377d88fdda; no voters:
I20260928 16:37:22.479266 11748 raft_consensus.cc:2781] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:37:22.479357 11748 raft_consensus.cc:482] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:37:22.479389 11748 raft_consensus.cc:3037] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:22.480374 11748 raft_consensus.cc:504] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } }
I20260928 16:37:22.480727 11748 leader_election.cc:288] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [CANDIDATE]: Term 1 election: Requested vote from peers f4821abf0a5b4e1c90d7ebcfb83e51d7 (127.27.122.194:41341), c2ff3fe984f54f04a56ed3377d88fdda (127.27.122.195:35433)
I20260928 16:37:22.481037 11541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da6291ac35442b5a41104adacabc757" candidate_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7"
I20260928 16:37:22.481086 11677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da6291ac35442b5a41104adacabc757" candidate_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2ff3fe984f54f04a56ed3377d88fdda"
I20260928 16:37:22.481161 11541 raft_consensus.cc:3037] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:22.481191 11677 raft_consensus.cc:3037] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:22.482427 11677 raft_consensus.cc:2454] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1461e8d6cf9b4e98bd773aff186b5a01 in term 1.
I20260928 16:37:22.482427 11541 raft_consensus.cc:2454] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1461e8d6cf9b4e98bd773aff186b5a01 in term 1.
I20260928 16:37:22.482714 11336 leader_election.cc:302] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1461e8d6cf9b4e98bd773aff186b5a01, c2ff3fe984f54f04a56ed3377d88fdda; no voters:
I20260928 16:37:22.482936 11748 raft_consensus.cc:2781] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:37:22.483161 11748 raft_consensus.cc:686] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 1 LEADER]: Becoming Leader. State: Replica: 1461e8d6cf9b4e98bd773aff186b5a01, State: Running, Role: LEADER
I20260928 16:37:22.483486 11748 consensus_queue.cc:237] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } }
I20260928 16:37:22.486723 11252 catalog_manager.cc:5495] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1461e8d6cf9b4e98bd773aff186b5a01 (127.27.122.193). New cstate: current_term: 1 leader_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } health_report { overall_health: UNKNOWN } } }
I20260928 16:37:22.505461 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.501820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.27.122.193:37977)
0928 16:37:22.501972 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:22.501977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:22.501991 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:37:22.502147 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:22.502149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:22.502201 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:22.502314 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:37:22.502591 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.503086 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.503947 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:22.504163 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:22.504274 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:22.504612 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:37:22.505066 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:37:22.505197 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:22.505260 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 16:37:22.505320 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:37:22.511209 11677 raft_consensus.cc:1261] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda [term 1 FOLLOWER]: Refusing update from remote peer 1461e8d6cf9b4e98bd773aff186b5a01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:37:22.511209 11541 raft_consensus.cc:1261] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 1 FOLLOWER]: Refusing update from remote peer 1461e8d6cf9b4e98bd773aff186b5a01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:37:22.511933 11748 consensus_queue.cc:1035] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:37:22.512149 11753 consensus_queue.cc:1035] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:37:22.520228 11758 mvcc.cc:205] Tried to move back new op lower bound from 7334352660518625280 to 7334352660414947328. Current Snapshot: MvccSnapshot[applied={T|T < 7334352660518625280}]
I20260928 16:37:22.525520 11757 mvcc.cc:205] Tried to move back new op lower bound from 7334352660518625280 to 7334352660414947328. Current Snapshot: MvccSnapshot[applied={T|T < 7334352660518625280 or (T in {7334352660518625280})}]
I20260928 16:37:22.527765 11253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:37:22.527951 11253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:37:22.530463 11253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Sending DeleteTablet for 3 replicas of tablet 4da6291ac35442b5a41104adacabc757
I20260928 16:37:22.531277 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 11316
I20260928 16:37:22.531309 11657 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da6291ac35442b5a41104adacabc757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:37:22 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:35864
I20260928 16:37:22.531966 11521 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da6291ac35442b5a41104adacabc757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:37:22 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:57260
I20260928 16:37:22.532631 11767 tablet_replica.cc:330] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7: stopping tablet replica
I20260928 16:37:22.532891 11767 raft_consensus.cc:2229] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:37:22.533248 11767 raft_consensus.cc:2258] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:37:22.534754 11767 ts_tablet_manager.cc:1905] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:37:22.537091 11766 tablet_replica.cc:330] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda: stopping tablet replica
I20260928 16:37:22.537366 11766 raft_consensus.cc:2229] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:37:22.537662 11766 raft_consensus.cc:2258] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:37:22.538415 11767 ts_tablet_manager.cc:1918] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:37:22.538534 11767 log.cc:1196] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/4da6291ac35442b5a41104adacabc757
I20260928 16:37:22.538944 11767 ts_tablet_manager.cc:1939] T 4da6291ac35442b5a41104adacabc757 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Deleting consensus metadata
I20260928 16:37:22.539032 11766 ts_tablet_manager.cc:1905] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 16:37:22.540206 11237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:37977 (error 108)
I20260928 16:37:22.540490 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 11455
I20260928 16:37:22.540699 11237 catalog_manager.cc:4841] TS f4821abf0a5b4e1c90d7ebcfb83e51d7 (127.27.122.194:41341): tablet 4da6291ac35442b5a41104adacabc757 (table test-table [id=14546e715d114bd4a7c82821dd63f35f]) successfully deleted
I20260928 16:37:22.542208 11766 ts_tablet_manager.cc:1918] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:37:22.542307 11766 log.cc:1196] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/4da6291ac35442b5a41104adacabc757
I20260928 16:37:22.542689 11766 ts_tablet_manager.cc:1939] T 4da6291ac35442b5a41104adacabc757 P c2ff3fe984f54f04a56ed3377d88fdda: Deleting consensus metadata
I20260928 16:37:22.543705 11240 catalog_manager.cc:4841] TS c2ff3fe984f54f04a56ed3377d88fdda (127.27.122.195:35433): tablet 4da6291ac35442b5a41104adacabc757 (table test-table [id=14546e715d114bd4a7c82821dd63f35f]) successfully deleted
I20260928 16:37:22.545393 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:22.545120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:37977 (local address 127.0.0.1:55594)
0928 16:37:22.545251 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:22.545321 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:37977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 16:37:22.545634 11237 catalog_manager.cc:4584] TS 1461e8d6cf9b4e98bd773aff186b5a01 (127.27.122.193:37977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4da6291ac35442b5a41104adacabc757: Network error: Client connection negotiation failed: client connection to 127.27.122.193:37977: connect: Connection refused (error 111)
I20260928 16:37:22.547324 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 11594
I20260928 16:37:22.553768 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 11221
I20260928 16:37:42.560492 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:38467
--webserver_interface=127.27.122.254
--webserver_port=43401
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:38467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:37:42.669243 11769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:42.669489 11769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:42.669528 11769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:42.673448 11769 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:37:42.673501 11769 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:37:42.673516 11769 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:37:42.673529 11769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:37:42.673547 11769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:42.673596 11769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:42.673628 11769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:37:42.673648 11769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:37:42.678843 11769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:38467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:38467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=43401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:42.679862 11769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:42.680869 11769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:42.693867 11776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:42.693907 11777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:42.693971 11769 server_base.cc:1034] running on GCE node
W20260928 16:37:42.693861 11779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:42.694631 11769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:42.695739 11769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:42.697096 11769 hybrid_clock.cc:648] HybridClock initialized: now 1790613462697060 us; error 58 us; skew 500 ppm
Sep 28 16:37:42 dist-test-slave-g9sh krb5kdc[11207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613462, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:42.699839 11769 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:37:42.700690 11769 webserver.cc:460] Webserver started at http://127.27.122.254:43401/ using document root <none> and password file <none>
I20260928 16:37:42.701306 11769 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:42.701416 11769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:42.705276 11769 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 16:37:42.707412 11786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:42.708564 11769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:37:42.708705 11769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "ccbef86f21314323a24936b454aa1ce4"
format_stamp: "Formatted at 2026-09-28 16:37:21 on dist-test-slave-g9sh"
server_key: "6a644e0d6760e1937867266f5ae50600"
server_key_iv: "945ba9fbd8628bfc062a4c25aeaaa81d"
server_key_version: "encryptionkey@0"
I20260928 16:37:42.709326 11769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:42.720293 11769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:42.723002 11769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:42.723193 11769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:42.730013 11769 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:38467
I20260928 16:37:42.730302 11839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:38467 every 8 connection(s)
I20260928 16:37:42.731110 11769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:37:42.735232 11840 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:37:42.736685 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 11769
I20260928 16:37:42.736788 11840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Bootstrap starting.
I20260928 16:37:42.737198 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:37977
--local_ip_for_outbound_sockets=127.27.122.193
--tserver_master_addrs=127.27.122.254:38467
--webserver_port=38881
--webserver_interface=127.27.122.193
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:37:42.753832 11840 log.cc:824] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:42.762458 11840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:37:42.762813 11840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Bootstrap complete.
I20260928 16:37:42.767021 11840 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } }
I20260928 16:37:42.767431 11840 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbef86f21314323a24936b454aa1ce4, State: Initialized, Role: FOLLOWER
I20260928 16:37:42.767954 11840 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } }
I20260928 16:37:42.768105 11840 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:37:42.768199 11840 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:37:42.768301 11840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:37:42.770175 11840 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } }
I20260928 16:37:42.770506 11840 leader_election.cc:302] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccbef86f21314323a24936b454aa1ce4; no voters:
I20260928 16:37:42.770800 11840 leader_election.cc:288] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 16:37:42.770931 11844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:37:42.771265 11844 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 2 LEADER]: Becoming Leader. State: Replica: ccbef86f21314323a24936b454aa1ce4, State: Running, Role: LEADER
I20260928 16:37:42.771625 11844 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } }
I20260928 16:37:42.771993 11840 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:37:42.772560 11844 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccbef86f21314323a24936b454aa1ce4. Latest consensus state: current_term: 2 leader_uuid: "ccbef86f21314323a24936b454aa1ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } } }
I20260928 16:37:42.772753 11844 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:37:42.773180 11846 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ccbef86f21314323a24936b454aa1ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } } }
I20260928 16:37:42.773286 11846 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:37:42.773469 11851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:37:42.778190 11851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:37:42.778815 11851 catalog_manager.cc:1259] Loaded cluster ID: 9fddab6d07004348b753b5e417948ea2
I20260928 16:37:42.778875 11851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:37:42.781311 11851 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:37:42.782171 11851 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Loaded TSK: 0
I20260928 16:37:42.795074 11851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Generated new TSK 1
I20260928 16:37:42.795300 11851 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:37:42.854187 11842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:42.854408 11842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:42.854452 11842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:42.858436 11842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:42.858484 11842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:42.858541 11842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:37:42.863545 11842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:37977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=38881
--tserver_master_addrs=127.27.122.254:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:42.864470 11842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:42.865482 11842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:42.875859 11867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:42.875815 11871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:42.875936 11869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:42.876183 11842 server_base.cc:1034] running on GCE node
I20260928 16:37:42.876695 11842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:42.877460 11842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:42.878671 11842 hybrid_clock.cc:648] HybridClock initialized: now 1790613462878635 us; error 46 us; skew 500 ppm
Sep 28 16:37:42 dist-test-slave-g9sh krb5kdc[11207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613462, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:42.881910 11842 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:37:42.882869 11842 webserver.cc:460] Webserver started at http://127.27.122.193:38881/ using document root <none> and password file <none>
I20260928 16:37:42.883412 11842 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:42.883484 11842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:42.887286 11842 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:37:42.889430 11878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:42.890689 11842 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:37:42.890779 11842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "1461e8d6cf9b4e98bd773aff186b5a01"
format_stamp: "Formatted at 2026-09-28 16:37:21 on dist-test-slave-g9sh"
server_key: "2622e725b6be020739ab2c608273574a"
server_key_iv: "bddd22a38adc7af75a31e31024d4f6c9"
server_key_version: "encryptionkey@0"
I20260928 16:37:42.891335 11842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:42.900477 11842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:42.903455 11842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:42.903640 11842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:42.904232 11842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:42.905377 11886 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:37:42.909235 11842 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:37:42.909297 11842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260928 16:37:42.909365 11842 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:37:42.910938 11842 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:37:42.910995 11842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 16:37:42.911247 11886 tablet_bootstrap.cc:492] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Bootstrap starting.
I20260928 16:37:42.921695 11842 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:37977
I20260928 16:37:42.921759 11993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:37977 every 8 connection(s)
I20260928 16:37:42.922892 11842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:37:42.923264 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 11842
I20260928 16:37:42.923789 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:41341
--local_ip_for_outbound_sockets=127.27.122.194
--tserver_master_addrs=127.27.122.254:38467
--webserver_port=42437
--webserver_interface=127.27.122.194
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:37:42.932560 11886 log.cc:824] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:37:42 dist-test-slave-g9sh krb5kdc[11207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613462, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:37:42.940033 11886 tablet_bootstrap.cc:492] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:37:42.940421 11886 tablet_bootstrap.cc:492] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Bootstrap complete.
I20260928 16:37:42.941820 11886 ts_tablet_manager.cc:1397] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.013s
I20260928 16:37:42.943130 11998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:42.925264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:43139)
0928 16:37:42.926081 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:42.926140 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:42.927068 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:42.928518 (+ 1450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:42.928527 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:42.929073 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:42.929802 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:42.929827 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:42.931838 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:42.931845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:42.933487 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:42.933502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:42.934467 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:42.934473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:42.934741 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:42.935484 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:42.935509 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:42.937395 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:42.939819 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:42.939824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:42.939837 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:42.940176 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:42.940572 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:42.940575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:42.940577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:42.940749 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:42.941417 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:42.941426 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:42.941719 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 16:37:42.941938 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 16:37:42.942254 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":144,"threads_started":1}
I20260928 16:37:42.943286 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:42.925248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:43139 (local address 127.27.122.254:38467)
0928 16:37:42.925755 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:42.925776 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:42.926271 (+ 495us) server_negotiation.cc:407] Connection header received
0928 16:37:42.927544 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:42.927566 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:42.927992 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:42.928412 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:42.930023 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:42.931677 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:42.933628 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:42.934304 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:42.934471 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:42.937609 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:42.937615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:42.937631 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:42.937665 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:42.939692 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:42.940301 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:42.940307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:42.940312 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:42.940456 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:42.940858 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:42.940861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:42.940863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:42.941298 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:42.941425 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:42.942077 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 16:37:42.942376 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":118,"threads_started":1}
I20260928 16:37:42.944619 11886 raft_consensus.cc:348] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } }
I20260928 16:37:42.945099 11886 raft_consensus.cc:729] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1461e8d6cf9b4e98bd773aff186b5a01, State: Initialized, Role: FOLLOWER
I20260928 16:37:42.945636 11886 consensus_queue.cc:260] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } }
I20260928 16:37:42.945937 11994 heartbeater.cc:347] Connected to a master server at 127.27.122.254:38467
I20260928 16:37:42.946161 11994 heartbeater.cc:464] Registering TS with master...
I20260928 16:37:42.946609 11886 ts_tablet_manager.cc:1428] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:37:42.946647 11994 heartbeater.cc:511] Master 127.27.122.254:38467 requested a full tablet report, sending...
I20260928 16:37:42.949126 11803 ts_manager.cc:194] Registered new tserver with Master: 1461e8d6cf9b4e98bd773aff186b5a01 (127.27.122.193:37977)
I20260928 16:37:42.952862 11803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:43139
I20260928 16:37:42.955626 11994 heartbeater.cc:503] Master 127.27.122.254:38467 was elected leader, sending a full tablet report...
Sep 28 16:37:42 dist-test-slave-g9sh krb5kdc[11207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613462, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for kudu/127.27.122.193@KRBTEST.COM
I20260928 16:37:42.962525 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:42.952257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:37977 (local address 127.0.0.1:39650)
0928 16:37:42.952977 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:42.953022 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:42.953164 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:42.953686 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:42.953694 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:42.954177 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:42.954284 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:42.954290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:42.955229 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:42.955234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:42.955866 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:42.955875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:42.956512 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:42.956515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:42.956626 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:42.957245 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:42.957256 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:42.958727 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:42.960706 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:42.960717 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:42.960722 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:42.961033 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:42.961349 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:42.961352 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:42.961354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:42.961441 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:42.961896 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:42.961903 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:42.962027 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 16:37:42.962168 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 16:37:42.962357 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":100,"threads_started":1}
I20260928 16:37:42.962566 12003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:42.952311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39650 (local address 127.27.122.193:37977)
0928 16:37:42.952677 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:42.952964 (+ 287us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:42.953096 (+ 132us) server_negotiation.cc:407] Connection header received
0928 16:37:42.953312 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:42.953317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:42.953500 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:42.954014 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:42.954399 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:42.955116 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:42.956021 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:42.956390 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:42.956510 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:42.958865 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:42.958876 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:42.958887 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:42.958958 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:42.960577 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:42.961155 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:42.961160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:42.961164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:42.961245 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:42.961548 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:42.961551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:42.961555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:42.961779 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:42.961921 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:42.962262 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 16:37:42.962419 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":280,"thread_start_us":90,"threads_started":1}
I20260928 16:37:42.963459 11928 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da6291ac35442b5a41104adacabc757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:37:22 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:39650
I20260928 16:37:42.964419 12004 tablet_replica.cc:330] stopping tablet replica
I20260928 16:37:42.964704 12004 raft_consensus.cc:2229] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:37:42.964984 12004 raft_consensus.cc:2258] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:37:42.965911 12004 ts_tablet_manager.cc:1905] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:37:42.969431 12004 ts_tablet_manager.cc:1918] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:37:42.969537 12004 log.cc:1196] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal/wals/4da6291ac35442b5a41104adacabc757
I20260928 16:37:42.969808 12004 ts_tablet_manager.cc:1939] T 4da6291ac35442b5a41104adacabc757 P 1461e8d6cf9b4e98bd773aff186b5a01: Deleting consensus metadata
I20260928 16:37:42.971333 11787 catalog_manager.cc:4841] TS 1461e8d6cf9b4e98bd773aff186b5a01 (127.27.122.193:37977): tablet 4da6291ac35442b5a41104adacabc757 (table test-table [id=14546e715d114bd4a7c82821dd63f35f]) successfully deleted
W20260928 16:37:43.042492 11996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:43.042706 11996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:43.042745 11996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:43.046832 11996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:43.046880 11996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:43.046965 11996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:37:43.051949 11996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:41341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=42437
--tserver_master_addrs=127.27.122.254:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:43.052883 11996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:43.053886 11996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:43.063207 12010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:43.063228 12009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:43.063213 12012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:43.063503 11996 server_base.cc:1034] running on GCE node
I20260928 16:37:43.063908 11996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:43.064499 11996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:43.065651 11996 hybrid_clock.cc:648] HybridClock initialized: now 1790613463065620 us; error 43 us; skew 500 ppm
Sep 28 16:37:43 dist-test-slave-g9sh krb5kdc[11207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613463, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:43.068188 11996 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:37:43.069079 11996 webserver.cc:460] Webserver started at http://127.27.122.194:42437/ using document root <none> and password file <none>
I20260928 16:37:43.069604 11996 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:43.069697 11996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:43.073387 11996 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 16:37:43.075456 12019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:43.076643 11996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:37:43.076764 11996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7"
format_stamp: "Formatted at 2026-09-28 16:37:22 on dist-test-slave-g9sh"
server_key: "cc056c90f6aa34eb8b5d026b4dbbbc7b"
server_key_iv: "98d08574fae11adc5af1b23593cc5867"
server_key_version: "encryptionkey@0"
I20260928 16:37:43.077351 11996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:43.094475 11996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:43.097828 11996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:43.098029 11996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:43.098599 11996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:43.099390 11996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:37:43.099436 11996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:43.099505 11996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:37:43.099541 11996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:43.107995 11996 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:41341
I20260928 16:37:43.108217 12134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:41341 every 8 connection(s)
I20260928 16:37:43.109242 11996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:37:43.111181 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 11996
I20260928 16:37:43.111606 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:35433
--local_ip_for_outbound_sockets=127.27.122.195
--tserver_master_addrs=127.27.122.254:38467
--webserver_port=36979
--webserver_interface=127.27.122.195
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:37:43 dist-test-slave-g9sh krb5kdc[11207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613463, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:37:43.124231 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.111065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:39895 (local address 127.27.122.254:38467)
0928 16:37:43.111227 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:43.111233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:43.112073 (+ 840us) server_negotiation.cc:407] Connection header received
0928 16:37:43.113872 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:43.113875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:43.113926 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:43.114003 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:43.115390 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.116012 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.117231 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.117465 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.117581 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.120242 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:43.120248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:43.120254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:43.120290 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:43.122178 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:43.122701 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:43.122705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:43.122707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:43.122814 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:43.123168 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:43.123173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:43.123178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:43.123396 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:43.123496 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:43.123934 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 16:37:43.124067 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:37:43.125044 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.111459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:39895)
0928 16:37:43.111925 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:43.111977 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:43.113625 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:43.114239 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:43.114250 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:43.114764 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:43.115237 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:43.115250 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.116145 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.116152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:43.117103 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:43.117114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.117616 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.117620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:43.117792 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.118377 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:43.118393 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:43.120066 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:43.122309 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:43.122313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:43.122323 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:43.122587 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:43.122919 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:43.122922 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:43.122923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:43.123065 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:43.123503 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:43.123507 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:43.123718 (+ 211us) client_negotiation.cc:769] Sending connection context
0928 16:37:43.123878 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 16:37:43.124119 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":135,"threads_started":1}
I20260928 16:37:43.126169 12135 heartbeater.cc:347] Connected to a master server at 127.27.122.254:38467
I20260928 16:37:43.126410 12135 heartbeater.cc:464] Registering TS with master...
I20260928 16:37:43.126892 12135 heartbeater.cc:511] Master 127.27.122.254:38467 requested a full tablet report, sending...
I20260928 16:37:43.127888 11803 ts_manager.cc:194] Registered new tserver with Master: f4821abf0a5b4e1c90d7ebcfb83e51d7 (127.27.122.194:41341)
I20260928 16:37:43.128449 11803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:39895
W20260928 16:37:43.224725 12139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:37:43.225080 12139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:37:43.225176 12139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:37:43.229089 12139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:37:43.229171 12139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:37:43.229259 12139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:37:43.234295 12139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:35433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=36979
--tserver_master_addrs=127.27.122.254:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:37:43.235275 12139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:37:43.236270 12139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:37:43.246119 12145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:37:43.246160 12144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:43.246201 12139 server_base.cc:1034] running on GCE node
W20260928 16:37:43.246160 12147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:37:43.246786 12139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:37:43.247382 12139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:37:43.248570 12139 hybrid_clock.cc:648] HybridClock initialized: now 1790613463248528 us; error 53 us; skew 500 ppm
Sep 28 16:37:43 dist-test-slave-g9sh krb5kdc[11207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613463, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:37:43.251216 12139 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:37:43.252035 12139 webserver.cc:460] Webserver started at http://127.27.122.195:36979/ using document root <none> and password file <none>
I20260928 16:37:43.252537 12139 fs_manager.cc:359] Metadata directory not provided
I20260928 16:37:43.252606 12139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:37:43.256161 12139 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:37:43.258235 12154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:43.259433 12139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:37:43.259577 12139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "c2ff3fe984f54f04a56ed3377d88fdda"
format_stamp: "Formatted at 2026-09-28 16:37:22 on dist-test-slave-g9sh"
server_key: "c3f15835e4b5f39b13cb9d74cb65dc7f"
server_key_iv: "cd7f9afa8586d2ef2949a2066399fe21"
server_key_version: "encryptionkey@0"
I20260928 16:37:43.260077 12139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:37:43.271787 12139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:37:43.274897 12139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:37:43.275049 12139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:37:43.275516 12139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:37:43.276242 12139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:37:43.276300 12139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:43.276338 12139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:37:43.276360 12139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:37:43.284750 12139 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:35433
I20260928 16:37:43.284821 12270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:35433 every 8 connection(s)
I20260928 16:37:43.285719 12139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:37:43.288398 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 12139
Sep 28 16:37:43 dist-test-slave-g9sh krb5kdc[11207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613463, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:37:43.299136 12274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.289090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60012 (local address 127.27.122.254:38467)
0928 16:37:43.289317 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:43.289320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:43.289622 (+ 302us) server_negotiation.cc:407] Connection header received
0928 16:37:43.289751 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:43.289753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:43.289795 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:43.289874 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:43.290819 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.291354 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.293473 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.293712 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.293824 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.295488 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:43.295491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:43.295493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:43.295517 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:43.297238 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:43.297729 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:43.297735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:43.297737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:43.297834 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:43.298184 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:43.298187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:43.298188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:43.298436 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:43.298687 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:43.298767 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 16:37:43.299005 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":97,"threads_started":1}
I20260928 16:37:43.301177 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.287419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:57179 (local address 127.27.122.254:38467)
0928 16:37:43.287549 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:43.287553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:43.288365 (+ 812us) server_negotiation.cc:407] Connection header received
0928 16:37:43.290058 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:43.290060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:43.290084 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:43.290126 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:43.291477 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.292226 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.293948 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.294249 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.294373 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.297097 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:43.297103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:43.297107 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:43.297153 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:43.298801 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:43.299340 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:43.299344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:43.299346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:43.299428 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:43.300014 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:43.300017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:43.300019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:43.300193 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:43.300306 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:43.300885 (+ 579us) server_negotiation.cc:299] Negotiation successful
0928 16:37:43.301038 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:37:43.301920 12273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.287798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:57179)
0928 16:37:43.288224 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:43.288264 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:43.289836 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:43.290352 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:43.290364 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:43.290861 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:37:43.291329 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:43.291343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.292377 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.292381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:43.293820 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:43.293831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.294429 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.294435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:43.294674 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.295228 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:37:43.295251 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:37:43.296876 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:37:43.298927 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:43.298933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:43.298948 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:43.299204 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:43.299557 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:37:43.299560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:37:43.299562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:37:43.299877 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:37:43.300319 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:37:43.300325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:37:43.300630 (+ 305us) client_negotiation.cc:769] Sending connection context
0928 16:37:43.300791 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 16:37:43.301105 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
I20260928 16:37:43.302951 12271 heartbeater.cc:347] Connected to a master server at 127.27.122.254:38467
I20260928 16:37:43.303184 12271 heartbeater.cc:464] Registering TS with master...
I20260928 16:37:43.303645 12271 heartbeater.cc:511] Master 127.27.122.254:38467 requested a full tablet report, sending...
I20260928 16:37:43.304596 11803 ts_manager.cc:194] Registered new tserver with Master: c2ff3fe984f54f04a56ed3377d88fdda (127.27.122.195:35433)
I20260928 16:37:43.305154 11803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:57179
I20260928 16:37:43.313035 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:37:43.316728 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.313550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60014 (local address 127.27.122.254:38467)
0928 16:37:43.313689 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:43.313693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:43.313874 (+ 181us) server_negotiation.cc:407] Connection header received
0928 16:37:43.313985 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:43.313987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:43.314071 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:43.314190 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:37:43.314408 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.314978 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.315765 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.316037 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.316117 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.316353 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:37:43.316464 (+ 111us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 16:37:43.316646 (+ 182us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60014: token expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 16:37:43.316802 11997 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60014: token expired
W20260928 16:37:43.317256 11729 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.122.254:38467)
I20260928 16:37:43.325856 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.317657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60022 (local address 127.27.122.254:38467)
0928 16:37:43.317772 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:43.317778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:43.317830 (+ 52us) server_negotiation.cc:407] Connection header received
0928 16:37:43.317988 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:43.317991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:43.318068 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:43.318149 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:37:43.318982 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.319673 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.320718 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.321062 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.321171 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.322547 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:37:43.322550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:37:43.322552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:37:43.322571 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:37:43.323970 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:43.324389 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:43.324396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:43.324400 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:43.324502 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:37:43.324838 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:37:43.324841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:37:43.324843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:37:43.325119 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:37:43.325265 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:43.325533 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 16:37:43.325662 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:37:43.328225 11803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:37:43.332628 11803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:37:43.340485 11928 tablet_service.cc:1467] Processing CreateTablet for tablet 8c46e34d1037476691afe24be2674731 (DEFAULT_TABLE table=test-table [id=ddc2dd9abc674f0a92c7a183a7900a0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:37:43.341197 11928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c46e34d1037476691afe24be2674731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:43.343652 12001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.339115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:35433 (local address 127.0.0.1:33338)
0928 16:37:43.339284 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:43.339298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:43.339371 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:43.340203 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:43.340205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:43.340223 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:43.340324 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:43.340330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.341473 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.341477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:43.342306 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:43.342315 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.343271 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.343275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:43.343371 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.343417 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 16:37:43.343465 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 16:37:43.343534 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 16:37:43.343899 11886 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01: Bootstrap starting.
I20260928 16:37:43.344340 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.339305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.27.122.195:35433)
0928 16:37:43.339691 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:43.339703 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:43.339727 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:37:43.339803 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:43.339808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:43.340038 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:43.340226 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:43.340420 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.341337 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.342430 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.343141 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.343921 (+ 780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.344068 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:43.344115 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 16:37:43.344201 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":157,"threads_started":1}
I20260928 16:37:43.344815 11886 tablet_bootstrap.cc:654] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:43.345327 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.339612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:41341 (local address 127.0.0.1:38212)
0928 16:37:43.339798 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:43.339812 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:43.339899 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:43.340711 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:43.340715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:43.340733 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:43.340812 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:43.340817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.342757 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.342760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:43.343843 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:43.343856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.344942 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.344946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:43.345035 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.345055 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 16:37:43.345138 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 16:37:43.345206 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":1,"thread_start_us":51,"threads_started":1}
I20260928 16:37:43.345391 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.339800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38212 (local address 127.27.122.194:41341)
0928 16:37:43.340247 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:43.340259 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:43.340280 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:37:43.340356 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:43.340361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:43.340557 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:43.340723 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:43.340923 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.342635 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.343980 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.344787 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.344946 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.345069 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:43.345170 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 16:37:43.345256 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":125,"threads_started":1}
I20260928 16:37:43.346554 11886 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01: No bootstrap required, opened a new log
I20260928 16:37:43.346632 11886 ts_tablet_manager.cc:1397] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 16:37:43.347052 11886 raft_consensus.cc:348] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:37:43.347159 11886 raft_consensus.cc:374] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:43.347196 11886 raft_consensus.cc:729] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1461e8d6cf9b4e98bd773aff186b5a01, State: Initialized, Role: FOLLOWER
I20260928 16:37:43.347321 11886 consensus_queue.cc:260] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:37:43.347579 11886 ts_tablet_manager.cc:1428] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 16:37:43.347739 12204 tablet_service.cc:1467] Processing CreateTablet for tablet 8c46e34d1037476691afe24be2674731 (DEFAULT_TABLE table=test-table [id=ddc2dd9abc674f0a92c7a183a7900a0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:37:43.348579 12204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c46e34d1037476691afe24be2674731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:43.351787 12068 tablet_service.cc:1467] Processing CreateTablet for tablet 8c46e34d1037476691afe24be2674731 (DEFAULT_TABLE table=test-table [id=ddc2dd9abc674f0a92c7a183a7900a0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:37:43.352574 12068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c46e34d1037476691afe24be2674731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:37:43.355793 12284 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Bootstrap starting.
I20260928 16:37:43.358071 12284 tablet_bootstrap.cc:654] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:43.358829 12285 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Bootstrap starting.
I20260928 16:37:43.359436 12284 log.cc:824] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:43.361523 12285 tablet_bootstrap.cc:654] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Neither blocks nor log segments found. Creating new log.
I20260928 16:37:43.361668 12284 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: No bootstrap required, opened a new log
I20260928 16:37:43.361871 12284 ts_tablet_manager.cc:1397] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260928 16:37:43.362358 12285 log.cc:824] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Log is configured to *not* fsync() on all Append() calls
I20260928 16:37:43.364322 12285 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7: No bootstrap required, opened a new log
I20260928 16:37:43.364495 12285 ts_tablet_manager.cc:1397] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:37:43.365645 12284 raft_consensus.cc:348] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:37:43.365923 12284 raft_consensus.cc:374] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:43.365988 12284 raft_consensus.cc:729] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ff3fe984f54f04a56ed3377d88fdda, State: Initialized, Role: FOLLOWER
I20260928 16:37:43.366430 12284 consensus_queue.cc:260] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:37:43.367065 12271 heartbeater.cc:503] Master 127.27.122.254:38467 was elected leader, sending a full tablet report...
I20260928 16:37:43.367430 12284 ts_tablet_manager.cc:1428] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260928 16:37:43.368139 12285 raft_consensus.cc:348] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:37:43.368368 12285 raft_consensus.cc:374] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:37:43.368427 12285 raft_consensus.cc:729] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4821abf0a5b4e1c90d7ebcfb83e51d7, State: Initialized, Role: FOLLOWER
I20260928 16:37:43.368781 12285 consensus_queue.cc:260] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:37:43.369437 12135 heartbeater.cc:503] Master 127.27.122.254:38467 was elected leader, sending a full tablet report...
I20260928 16:37:43.369772 12285 ts_tablet_manager.cc:1428] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260928 16:37:43.424489 11995 tablet.cc:2369] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:37:43.534973 12288 raft_consensus.cc:482] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:37:43.535218 12288 raft_consensus.cc:504] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:37:43.536612 12288 leader_election.cc:288] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4821abf0a5b4e1c90d7ebcfb83e51d7 (127.27.122.194:41341), 1461e8d6cf9b4e98bd773aff186b5a01 (127.27.122.193:37977)
W20260928 16:37:43.536649 12272 tablet.cc:2369] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:37:43.541005 12273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.536842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:41341 (local address 127.27.122.195:59471)
0928 16:37:43.537029 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:43.537046 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:43.537150 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:43.537527 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:43.537531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:43.537557 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:43.537692 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:43.537699 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.538652 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.538656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:43.539698 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:43.539708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.540527 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.540531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:43.540648 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.540675 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:37:43.540758 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 16:37:43.540831 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 16:37:43.541007 12282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.537036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:59471 (local address 127.27.122.194:41341)
0928 16:37:43.537182 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:43.537187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:43.537223 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:37:43.537291 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:43.537297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:43.537365 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:43.537449 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:43.537839 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.538533 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.539840 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.540403 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.540500 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.540540 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:43.540785 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 16:37:43.540846 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:37:43.541605 12290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.537190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:37977 (local address 127.27.122.195:49407)
0928 16:37:43.537661 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:43.537680 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:37:43.537779 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:37:43.538072 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:37:43.538075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:37:43.538095 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:37:43.538205 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:43.538211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.539204 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.539207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:43.540139 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:37:43.540149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.541113 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.541117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:37:43.541223 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.541258 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:37:43.541353 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:37:43.541412 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":154,"threads_started":1}
I20260928 16:37:43.541790 12089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c46e34d1037476691afe24be2674731" candidate_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" is_pre_election: true
I20260928 16:37:43.541935 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.537493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:49407 (local address 127.27.122.193:37977)
0928 16:37:43.537763 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:43.537770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:43.537789 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:37:43.537878 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:43.537880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:43.537937 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:43.538025 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:37:43.538324 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.539063 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.540272 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.540969 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.541099 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.541278 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:43.541495 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 16:37:43.541617 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260928 16:37:43.542110 12089 raft_consensus.cc:2454] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ff3fe984f54f04a56ed3377d88fdda in term 0.
I20260928 16:37:43.542526 11948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c46e34d1037476691afe24be2674731" candidate_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" is_pre_election: true
I20260928 16:37:43.542717 12155 leader_election.cc:302] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ff3fe984f54f04a56ed3377d88fdda, f4821abf0a5b4e1c90d7ebcfb83e51d7; no voters:
I20260928 16:37:43.542843 11948 raft_consensus.cc:2454] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ff3fe984f54f04a56ed3377d88fdda in term 0.
I20260928 16:37:43.542994 12288 raft_consensus.cc:2781] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:37:43.543097 12288 raft_consensus.cc:482] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:37:43.543154 12288 raft_consensus.cc:3037] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:43.544174 12288 raft_consensus.cc:504] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:37:43.544454 12288 leader_election.cc:288] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [CANDIDATE]: Term 1 election: Requested vote from peers f4821abf0a5b4e1c90d7ebcfb83e51d7 (127.27.122.194:41341), 1461e8d6cf9b4e98bd773aff186b5a01 (127.27.122.193:37977)
I20260928 16:37:43.544873 12089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c46e34d1037476691afe24be2674731" candidate_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7"
I20260928 16:37:43.544884 11948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c46e34d1037476691afe24be2674731" candidate_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1461e8d6cf9b4e98bd773aff186b5a01"
I20260928 16:37:43.545023 11948 raft_consensus.cc:3037] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:43.545025 12089 raft_consensus.cc:3037] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:37:43.546312 11948 raft_consensus.cc:2454] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ff3fe984f54f04a56ed3377d88fdda in term 1.
I20260928 16:37:43.546305 12089 raft_consensus.cc:2454] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ff3fe984f54f04a56ed3377d88fdda in term 1.
I20260928 16:37:43.546589 12155 leader_election.cc:302] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ff3fe984f54f04a56ed3377d88fdda, f4821abf0a5b4e1c90d7ebcfb83e51d7; no voters:
I20260928 16:37:43.546770 12288 raft_consensus.cc:2781] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:37:43.547057 12288 raft_consensus.cc:686] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 1 LEADER]: Becoming Leader. State: Replica: c2ff3fe984f54f04a56ed3377d88fdda, State: Running, Role: LEADER
I20260928 16:37:43.547379 12288 consensus_queue.cc:237] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:37:43.550500 11803 catalog_manager.cc:5495] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ff3fe984f54f04a56ed3377d88fdda (127.27.122.195). New cstate: current_term: 1 leader_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } health_report { overall_health: UNKNOWN } } }
I20260928 16:37:43.582451 12280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.578951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.27.122.195:35433)
0928 16:37:43.579100 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 16:37:43.579104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:37:43.579118 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:37:43.579213 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:37:43.579215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:37:43.579273 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:37:43.579371 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:37:43.579609 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.580121 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.581015 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:37:43.581224 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:37:43.581304 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:37:43.581572 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:37:43.582071 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:37:43.582163 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:37:43.582225 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 16:37:43.582275 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:37:43.588245 11948 raft_consensus.cc:1261] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 1 FOLLOWER]: Refusing update from remote peer c2ff3fe984f54f04a56ed3377d88fdda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:37:43.588356 12089 raft_consensus.cc:1261] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 1 FOLLOWER]: Refusing update from remote peer c2ff3fe984f54f04a56ed3377d88fdda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:37:43.588984 12288 consensus_queue.cc:1035] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [LEADER]: Connected to new peer: Peer: permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:37:43.589210 12292 consensus_queue.cc:1035] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:37:43.605468 11803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:37:43.605602 11803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:37:43.607833 11803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Sending DeleteTablet for 3 replicas of tablet 8c46e34d1037476691afe24be2674731
I20260928 16:37:43.608422 12204 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c46e34d1037476691afe24be2674731 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:37:43 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:33338
I20260928 16:37:43.608659 11928 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c46e34d1037476691afe24be2674731 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:37:43 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:39650
I20260928 16:37:43.608621 12068 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c46e34d1037476691afe24be2674731 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:37:43 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:38212
I20260928 16:37:43.608994 12305 tablet_replica.cc:330] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: stopping tablet replica
I20260928 16:37:43.609001 12306 tablet_replica.cc:330] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01: stopping tablet replica
I20260928 16:37:43.609066 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 11842
I20260928 16:37:43.609347 12305 raft_consensus.cc:2229] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:37:43.609962 12305 raft_consensus.cc:2258] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:37:43.611318 12305 ts_tablet_manager.cc:1905] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:37:43.614233 12305 ts_tablet_manager.cc:1918] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:37:43.614336 12305 log.cc:1196] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/8c46e34d1037476691afe24be2674731
I20260928 16:37:43.614682 12305 ts_tablet_manager.cc:1939] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Deleting consensus metadata
I20260928 16:37:43.615933 11790 catalog_manager.cc:4841] TS c2ff3fe984f54f04a56ed3377d88fdda (127.27.122.195:35433): tablet 8c46e34d1037476691afe24be2674731 (table test-table [id=ddc2dd9abc674f0a92c7a183a7900a0a]) successfully deleted
W20260928 16:37:43.616392 11787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:37977 (error 108)
I20260928 16:37:43.616565 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 11996
I20260928 16:37:43.617617 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.617327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:37977 (local address 127.0.0.1:39664)
0928 16:37:43.617477 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:43.617541 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:37977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260928 16:37:43.617769 11787 catalog_manager.cc:4584] TS 1461e8d6cf9b4e98bd773aff186b5a01 (127.27.122.193:37977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c46e34d1037476691afe24be2674731: Network error: Client connection negotiation failed: client connection to 127.27.122.193:37977: connect: Connection refused (error 111)
I20260928 16:37:43.625420 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 12139
I20260928 16:37:43.626228 12281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:37:43.625458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:41341 (local address 127.0.0.1:38228)
0928 16:37:43.626102 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 16:37:43.626157 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.194:41341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576}
I20260928 16:37:43.631599 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 11769
I20260928 16:38:03.638422 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:38467
--webserver_interface=127.27.122.254
--webserver_port=43401
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:38467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:03.747453 12309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:03.747759 12309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:03.747840 12309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:03.752100 12309 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 16:38:03.752203 12309 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 16:38:03.752236 12309 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 16:38:03.752269 12309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:03.752310 12309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:03.752352 12309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:03.752389 12309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:03.752421 12309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:03.757901 12309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:38467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:38467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=43401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:03.758976 12309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:03.760031 12309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:03.768489 12314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:03.768522 12315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:03.768515 12317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:03.768560 12309 server_base.cc:1034] running on GCE node
I20260928 16:38:03.769325 12309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:03.770352 12309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:03.771535 12309 hybrid_clock.cc:648] HybridClock initialized: now 1790613483771508 us; error 43 us; skew 500 ppm
Sep 28 16:38:03 dist-test-slave-g9sh krb5kdc[11207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613483, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:03.774293 12309 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:38:03.775072 12309 webserver.cc:460] Webserver started at http://127.27.122.254:43401/ using document root <none> and password file <none>
I20260928 16:38:03.775564 12309 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:03.775621 12309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:03.779397 12309 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:38:03.781504 12324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:03.782779 12309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:03.782909 12309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "ccbef86f21314323a24936b454aa1ce4"
format_stamp: "Formatted at 2026-09-28 16:37:21 on dist-test-slave-g9sh"
server_key: "6a644e0d6760e1937867266f5ae50600"
server_key_iv: "945ba9fbd8628bfc062a4c25aeaaa81d"
server_key_version: "encryptionkey@0"
I20260928 16:38:03.783443 12309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:03.793725 12309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:03.796350 12309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:03.796540 12309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:03.803452 12309 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:38467
I20260928 16:38:03.803717 12378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:38467 every 8 connection(s)
I20260928 16:38:03.804471 12309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:03.808640 12379 sys_catalog.cc:263] Verifying existing consensus state
I20260928 16:38:03.810240 12379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Bootstrap starting.
I20260928 16:38:03.814711 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 12309
I20260928 16:38:03.815112 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:37977
--local_ip_for_outbound_sockets=127.27.122.193
--tserver_master_addrs=127.27.122.254:38467
--webserver_port=38881
--webserver_interface=127.27.122.193
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:38:03.829208 12379 log.cc:824] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:03.840260 12379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:38:03.840636 12379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: Bootstrap complete.
I20260928 16:38:03.844674 12379 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } }
I20260928 16:38:03.845150 12379 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbef86f21314323a24936b454aa1ce4, State: Initialized, Role: FOLLOWER
I20260928 16:38:03.845690 12379 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } }
I20260928 16:38:03.845835 12379 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:03.845923 12379 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:03.846030 12379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 2 FOLLOWER]: Advancing to term 3
I20260928 16:38:03.847828 12379 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } }
I20260928 16:38:03.848163 12379 leader_election.cc:302] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccbef86f21314323a24936b454aa1ce4; no voters:
I20260928 16:38:03.848479 12379 leader_election.cc:288] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 16:38:03.848618 12384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 16:38:03.848865 12384 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [term 3 LEADER]: Becoming Leader. State: Replica: ccbef86f21314323a24936b454aa1ce4, State: Running, Role: LEADER
I20260928 16:38:03.849267 12384 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } }
I20260928 16:38:03.849692 12379 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:03.850749 12385 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ccbef86f21314323a24936b454aa1ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } } }
I20260928 16:38:03.850867 12385 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:03.851550 12386 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccbef86f21314323a24936b454aa1ce4. Latest consensus state: current_term: 3 leader_uuid: "ccbef86f21314323a24936b454aa1ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccbef86f21314323a24936b454aa1ce4" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38467 } } }
I20260928 16:38:03.851629 12386 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:03.854652 12402 catalog_manager.cc:1259] Loaded cluster ID: 9fddab6d07004348b753b5e417948ea2
I20260928 16:38:03.854708 12402 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: loading cluster ID for follower catalog manager: success
I20260928 16:38:03.857236 12402 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccbef86f21314323a24936b454aa1ce4: acquiring CA information for follower catalog manager: success
F20260928 16:38:03.858417 12402 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9bd86d59ad google::LogMessage::Fail() at ??:0
@ 0x7f9bd86d888a google::LogMessage::SendToLog() at ??:0
@ 0x7f9bd86d54de google::LogMessage::Flush() at ??:0
@ 0x7f9bd86d6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9bdb39918d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9be303e989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f9be301b023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f9be301b54b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f9be3014652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f9be30139e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7f9be305a541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9bda918512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9bdbf5c6db start_thread at ??:0
@ 0x7f9bd71d361f clone at ??:0
W20260928 16:38:03.947692 12381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:03.947923 12381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:03.947963 12381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:03.951937 12381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:03.951988 12381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:03.952044 12381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:38:03.957696 12381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:37977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=38881
--tserver_master_addrs=127.27.122.254:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:03.958683 12381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:03.959676 12381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:03.968997 12422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:03.969063 12423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:03.969063 12425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:03.969322 12381 server_base.cc:1034] running on GCE node
I20260928 16:38:03.969727 12381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:03.970355 12381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:03.971531 12381 hybrid_clock.cc:648] HybridClock initialized: now 1790613483971503 us; error 43 us; skew 500 ppm
Sep 28 16:38:03 dist-test-slave-g9sh krb5kdc[11207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613483, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:03.974402 12381 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:38:03.975224 12381 webserver.cc:460] Webserver started at http://127.27.122.193:38881/ using document root <none> and password file <none>
I20260928 16:38:03.975770 12381 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:03.975863 12381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:03.979811 12381 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260928 16:38:03.982062 12432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:03.983361 12381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 16:38:03.983515 12381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "1461e8d6cf9b4e98bd773aff186b5a01"
format_stamp: "Formatted at 2026-09-28 16:37:21 on dist-test-slave-g9sh"
server_key: "2622e725b6be020739ab2c608273574a"
server_key_iv: "bddd22a38adc7af75a31e31024d4f6c9"
server_key_version: "encryptionkey@0"
I20260928 16:38:03.984064 12381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:04.022935 12381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:04.026252 12381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:04.026451 12381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:04.027064 12381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:04.028191 12440 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:38:04.033748 12381 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:38:04.033838 12381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 16:38:04.033924 12381 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:38:04.035575 12381 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:38:04.035646 12381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 16:38:04.035948 12440 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01: Bootstrap starting.
I20260928 16:38:04.046617 12381 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:37977
I20260928 16:38:04.046838 12550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:37977 every 8 connection(s)
I20260928 16:38:04.047592 12381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:38:04.052320 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 12381
I20260928 16:38:04.052793 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:41341
--local_ip_for_outbound_sockets=127.27.122.194
--tserver_master_addrs=127.27.122.254:38467
--webserver_port=42437
--webserver_interface=127.27.122.194
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:38:04.058445 12440 log.cc:824] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01: Log is configured to *not* fsync() on all Append() calls
W20260928 16:38:04.061708 12553 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:04.049764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:32953)
0928 16:38:04.050278 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:04.050321 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:04.051092 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:04.060711 (+ 9619us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":151,"threads_started":1}
W20260928 16:38:04.062296 12551 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:38467 (0 consecutive failures): Network error: Failed to ping master at 127.27.122.254:38467: Client connection negotiation failed: client connection to 127.27.122.254:38467: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 16:38:04.062966 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:04.062708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:36875)
0928 16:38:04.062839 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:04.062905 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:38:04.063836 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:04.063600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:36739)
0928 16:38:04.063727 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:04.063778 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 16:38:04.064057 12551 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:38:04.066030 12440 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:38:04.066380 12440 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01: Bootstrap complete.
I20260928 16:38:04.067516 12440 ts_tablet_manager.cc:1397] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.004s
I20260928 16:38:04.069866 12440 raft_consensus.cc:348] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:38:04.070252 12440 raft_consensus.cc:729] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1461e8d6cf9b4e98bd773aff186b5a01, State: Initialized, Role: FOLLOWER
I20260928 16:38:04.070746 12440 consensus_queue.cc:260] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:38:04.071671 12440 ts_tablet_manager.cc:1428] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260928 16:38:04.071841 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:04.071604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:43295)
0928 16:38:04.071730 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:04.071784 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 16:38:04.072427 12553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:04.072221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:58441)
0928 16:38:04.072328 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:04.072373 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 16:38:04.164238 12554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:04.164515 12554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:04.164598 12554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:04.168556 12554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:04.168643 12554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:04.168730 12554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:38:04.173892 12554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:41341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=42437
--tserver_master_addrs=127.27.122.254:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:04.174885 12554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:04.175930 12554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:04.185377 12562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:04.185390 12554 server_base.cc:1034] running on GCE node
W20260928 16:38:04.185390 12561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:04.185397 12564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:04.186085 12554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:04.186677 12554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:04.187844 12554 hybrid_clock.cc:648] HybridClock initialized: now 1790613484187809 us; error 45 us; skew 500 ppm
Sep 28 16:38:04 dist-test-slave-g9sh krb5kdc[11207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613484, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:04.190783 12554 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:38:04.191639 12554 webserver.cc:460] Webserver started at http://127.27.122.194:42437/ using document root <none> and password file <none>
I20260928 16:38:04.192162 12554 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:04.192220 12554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:04.196167 12554 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:38:04.198390 12571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:04.199750 12554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:38:04.199872 12554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7"
format_stamp: "Formatted at 2026-09-28 16:37:22 on dist-test-slave-g9sh"
server_key: "cc056c90f6aa34eb8b5d026b4dbbbc7b"
server_key_iv: "98d08574fae11adc5af1b23593cc5867"
server_key_version: "encryptionkey@0"
I20260928 16:38:04.200389 12554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:04.220971 12554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:04.224252 12554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:04.224470 12554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:04.225101 12554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:04.226214 12578 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 16:38:04.230408 12554 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 16:38:04.230523 12554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 16:38:04.230628 12554 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 16:38:04.232295 12554 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 16:38:04.232363 12554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 16:38:04.232729 12578 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Bootstrap starting.
I20260928 16:38:04.243098 12554 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:41341
I20260928 16:38:04.243216 12690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:41341 every 8 connection(s)
I20260928 16:38:04.244065 12554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:38:04.247728 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:04.246023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:38763)
0928 16:38:04.246547 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:04.246682 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":121,"threads_started":1}
W20260928 16:38:04.248144 12691 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:38467 (0 consecutive failures): Network error: Failed to ping master at 127.27.122.254:38467: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
I20260928 16:38:04.248925 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:04.248587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:50841)
0928 16:38:04.248736 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:04.248803 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 16:38:04.249563 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:04.249341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:41465)
0928 16:38:04.249454 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:04.249504 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:38:04.249761 12691 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:38:04.250447 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 12554
I20260928 16:38:04.250948 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:35433
--local_ip_for_outbound_sockets=127.27.122.195
--tserver_master_addrs=127.27.122.254:38467
--webserver_port=36979
--webserver_interface=127.27.122.195
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:38:04.252660 12578 log.cc:824] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:04.259411 12578 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:38:04.259769 12578 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Bootstrap complete.
I20260928 16:38:04.260882 12578 ts_tablet_manager.cc:1397] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20260928 16:38:04.263444 12578 raft_consensus.cc:348] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:38:04.263839 12578 raft_consensus.cc:729] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4821abf0a5b4e1c90d7ebcfb83e51d7, State: Initialized, Role: FOLLOWER
I20260928 16:38:04.264334 12578 consensus_queue.cc:260] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:38:04.265316 12578 ts_tablet_manager.cc:1428] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260928 16:38:04.265487 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:04.265232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:36575)
0928 16:38:04.265366 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:04.265423 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 16:38:04.266117 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:04.265905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:35791)
0928 16:38:04.266012 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:04.266071 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 16:38:04.360457 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:04.360731 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:04.360808 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:04.364691 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:04.364764 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:04.364850 12694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:38:04.369917 12694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:35433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=36979
--tserver_master_addrs=127.27.122.254:38467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:04.370841 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:04.371780 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:04.380785 12702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:04.380939 12703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:04.380827 12705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:04.380987 12694 server_base.cc:1034] running on GCE node
I20260928 16:38:04.381362 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:04.381973 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:04.383162 12694 hybrid_clock.cc:648] HybridClock initialized: now 1790613484383150 us; error 24 us; skew 500 ppm
Sep 28 16:38:04 dist-test-slave-g9sh krb5kdc[11207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613484, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:04.392683 12694 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:38:04.393836 12694 webserver.cc:460] Webserver started at http://127.27.122.195:36979/ using document root <none> and password file <none>
I20260928 16:38:04.394547 12694 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:04.394640 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:04.398713 12694 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 16:38:04.400681 12712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:04.401975 12694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:38:04.402113 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "c2ff3fe984f54f04a56ed3377d88fdda"
format_stamp: "Formatted at 2026-09-28 16:37:22 on dist-test-slave-g9sh"
server_key: "c3f15835e4b5f39b13cb9d74cb65dc7f"
server_key_iv: "cd7f9afa8586d2ef2949a2066399fe21"
server_key_version: "encryptionkey@0"
I20260928 16:38:04.402642 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:04.415777 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:04.418864 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:04.419059 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:04.419617 12694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:04.420674 12694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:04.420756 12694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:04.420830 12694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:04.420881 12694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:04.429836 12694 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:35433
I20260928 16:38:04.429872 12828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:35433 every 8 connection(s)
I20260928 16:38:04.430780 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:38:04.434281 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:04.432730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:46793)
0928 16:38:04.433188 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:04.433614 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":133,"threads_started":1}
W20260928 16:38:04.434754 12829 heartbeater.cc:644] Failed to heartbeat to 127.27.122.254:38467 (0 consecutive failures): Network error: Failed to ping master at 127.27.122.254:38467: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
I20260928 16:38:04.435298 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:04.435092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:45979)
0928 16:38:04.435193 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:04.435249 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 16:38:04.435927 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:04.435752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:52875)
0928 16:38:04.435845 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:04.435885 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 16:38:04.436132 12829 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 16:38:04.437388 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 12694
I20260928 16:38:05.073586 12833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:05.073164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:48441)
0928 16:38:05.073418 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:05.073474 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260928 16:38:05.267159 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:05.266744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:51669)
0928 16:38:05.267002 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:05.267071 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260928 16:38:05.437068 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:05.436614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:35373)
0928 16:38:05.436849 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:05.436960 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":67,"threads_started":1}
I20260928 16:38:05.633518 12836 raft_consensus.cc:482] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:38:05.633750 12836 raft_consensus.cc:504] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:38:05.634869 12836 leader_election.cc:288] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c2ff3fe984f54f04a56ed3377d88fdda (127.27.122.195:35433), 1461e8d6cf9b4e98bd773aff186b5a01 (127.27.122.193:37977)
Sep 28 16:38:05 dist-test-slave-g9sh krb5kdc[11207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613484, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.193@KRBTEST.COM
Sep 28 16:38:05 dist-test-slave-g9sh krb5kdc[11207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613484, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.195@KRBTEST.COM
I20260928 16:38:05.648023 12839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:05.635391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:59561 (local address 127.27.122.193:37977)
0928 16:38:05.635813 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:05.635820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:05.635847 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:38:05.636240 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:05.636258 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:05.636463 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:05.636679 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:05.637975 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:05.639084 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:05.640226 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:05.640685 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:05.640859 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:05.643475 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:05.643481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:05.643492 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:05.643519 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:05.645443 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:05.646048 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:05.646054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:05.646059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:05.646186 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:05.646551 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:05.646556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:05.646558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:05.646917 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:05.647065 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:05.647579 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 16:38:05.647808 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":91,"threads_started":1}
I20260928 16:38:05.648026 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:05.635336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:44273 (local address 127.27.122.195:35433)
0928 16:38:05.635769 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:05.635794 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:05.635837 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:38:05.636717 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:05.636723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:05.637066 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:05.637330 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:05.637982 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:05.639078 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:05.640214 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:05.640681 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:05.640856 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:05.643801 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:05.643805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:05.643814 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:05.643839 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:05.645519 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:05.646052 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:05.646058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:05.646076 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:05.646184 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:05.646536 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:05.646541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:05.646543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:05.646916 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:05.647058 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:05.647572 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 16:38:05.647837 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":92,"threads_started":1}
I20260928 16:38:05.648034 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:05.635136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:37977 (local address 127.27.122.194:59561)
0928 16:38:05.635561 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:05.635576 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:05.635897 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:05.636765 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:05.636774 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:05.637295 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:05.637794 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:05.637811 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:05.639244 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:05.639248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:05.640097 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:05.640108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:05.640810 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:05.640814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:05.641038 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:05.641745 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:05.641758 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:05.643281 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:05.645579 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:05.645584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:05.645597 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:05.645924 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:05.646285 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:05.646289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:05.646291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:05.646453 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:05.647045 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:05.647049 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:05.647270 (+ 221us) client_negotiation.cc:769] Sending connection context
0928 16:38:05.647453 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 16:38:05.647803 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":220,"spinlock_wait_cycles":24448,"thread_start_us":90,"threads_started":1}
I20260928 16:38:05.648059 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:05.635136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:35433 (local address 127.27.122.194:44273)
0928 16:38:05.635345 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:05.635379 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:05.635891 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:05.637282 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:05.637285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:05.637473 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:05.637796 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:05.637814 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:05.639243 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:05.639245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:05.640097 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:05.640106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:05.640804 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:05.640807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:05.641035 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:05.641691 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:05.641708 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:05.643612 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:05.645632 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:05.645637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:05.645639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:05.645940 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:05.646284 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:05.646287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:05.646289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:05.646441 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:05.647034 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:05.647040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:05.647271 (+ 231us) client_negotiation.cc:769] Sending connection context
0928 16:38:05.647452 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 16:38:05.647888 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":202,"spinlock_wait_cycles":52352}
I20260928 16:38:05.649484 12505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c46e34d1037476691afe24be2674731" candidate_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" is_pre_election: true
I20260928 16:38:05.649824 12505 raft_consensus.cc:2454] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4821abf0a5b4e1c90d7ebcfb83e51d7 in term 1.
I20260928 16:38:05.650802 12572 leader_election.cc:302] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1461e8d6cf9b4e98bd773aff186b5a01, f4821abf0a5b4e1c90d7ebcfb83e51d7; no voters:
W20260928 16:38:05.651021 12575 leader_election.cc:341] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c2ff3fe984f54f04a56ed3377d88fdda (127.27.122.195:35433): Not found: Tablet not found: 8c46e34d1037476691afe24be2674731
I20260928 16:38:05.651115 12836 raft_consensus.cc:2781] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 16:38:05.651185 12836 raft_consensus.cc:482] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:38:05.651227 12836 raft_consensus.cc:3037] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:38:05.653375 12836 raft_consensus.cc:504] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:38:05.653745 12836 leader_election.cc:288] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [CANDIDATE]: Term 2 election: Requested vote from peers c2ff3fe984f54f04a56ed3377d88fdda (127.27.122.195:35433), 1461e8d6cf9b4e98bd773aff186b5a01 (127.27.122.193:37977)
I20260928 16:38:05.654290 12505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c46e34d1037476691afe24be2674731" candidate_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1461e8d6cf9b4e98bd773aff186b5a01"
W20260928 16:38:05.654362 12575 leader_election.cc:341] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer c2ff3fe984f54f04a56ed3377d88fdda (127.27.122.195:35433): Not found: Tablet not found: 8c46e34d1037476691afe24be2674731
I20260928 16:38:05.654440 12505 raft_consensus.cc:3037] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 1 FOLLOWER]: Advancing to term 2
I20260928 16:38:05.656319 12505 raft_consensus.cc:2454] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4821abf0a5b4e1c90d7ebcfb83e51d7 in term 2.
I20260928 16:38:05.656644 12572 leader_election.cc:302] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1461e8d6cf9b4e98bd773aff186b5a01, f4821abf0a5b4e1c90d7ebcfb83e51d7; no voters: c2ff3fe984f54f04a56ed3377d88fdda
I20260928 16:38:05.656847 12836 raft_consensus.cc:2781] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 16:38:05.657145 12836 raft_consensus.cc:686] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [term 2 LEADER]: Becoming Leader. State: Replica: f4821abf0a5b4e1c90d7ebcfb83e51d7, State: Running, Role: LEADER
I20260928 16:38:05.657461 12836 consensus_queue.cc:237] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:38:05.657781 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:05.657545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:51387)
0928 16:38:05.657662 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:05.657716 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 16:38:06.074888 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:06.074297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:36427)
0928 16:38:06.074660 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:06.074746 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":83,"threads_started":1}
W20260928 16:38:06.082916 12575 consensus_peers.cc:487] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 -> Peer c2ff3fe984f54f04a56ed3377d88fdda (127.27.122.195:35433): Couldn't send request to peer c2ff3fe984f54f04a56ed3377d88fdda. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c46e34d1037476691afe24be2674731. This is attempt 1: this message will repeat every 5th retry.
I20260928 16:38:06.191039 12505 raft_consensus.cc:1261] T 8c46e34d1037476691afe24be2674731 P 1461e8d6cf9b4e98bd773aff186b5a01 [term 2 FOLLOWER]: Refusing update from remote peer f4821abf0a5b4e1c90d7ebcfb83e51d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 16:38:06.191895 12836 consensus_queue.cc:1035] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 16:38:06.194167 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:06.193833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:39641)
0928 16:38:06.194007 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:06.194078 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 16:38:06.196717 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:06.196318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:37245)
0928 16:38:06.196564 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:06.196625 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":64,"threads_started":1}
I20260928 16:38:06.196871 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:06.196583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:39389)
0928 16:38:06.196733 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:06.196784 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 16:38:06.438171 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:06.437800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:33513)
0928 16:38:06.438026 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:06.438087 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":70,"threads_started":1}
I20260928 16:38:06.622622 12850 ts_tablet_manager.cc:927] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Initiating tablet copy from peer f4821abf0a5b4e1c90d7ebcfb83e51d7 (127.27.122.194:41341)
I20260928 16:38:06.623270 12850 tablet_copy_client.cc:316] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: tablet copy: Beginning tablet copy session from remote peer at address 127.27.122.194:41341
Sep 28 16:38:06 dist-test-slave-g9sh krb5kdc[11207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613484, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.194@KRBTEST.COM
I20260928 16:38:06.633301 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:06.623818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:41341 (local address 127.27.122.195:52371)
0928 16:38:06.624000 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:06.624019 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:06.624130 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:06.624700 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:06.624706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:06.625157 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:06.625303 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:06.625309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:06.626305 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:06.626308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:06.626951 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:06.626959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:06.627550 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:06.627553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:06.627661 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:06.628350 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:06.628362 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:06.629796 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:06.631729 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:06.631738 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:06.631743 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:06.631959 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:06.632244 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:06.632246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:06.632247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:06.632323 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:06.632749 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:06.632754 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:06.632859 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 16:38:06.633033 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 16:38:06.633166 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260928 16:38:06.633443 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:06.623942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:52371 (local address 127.27.122.194:41341)
0928 16:38:06.624273 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:06.624283 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:06.624303 (+ 20us) server_negotiation.cc:407] Connection header received
0928 16:38:06.624346 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:06.624350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:06.624534 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:06.624668 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:06.625430 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:06.626210 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:06.627059 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:06.627415 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:06.627500 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:06.629924 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:06.629932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:06.629944 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:06.629968 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:06.631626 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:06.632077 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:06.632082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:06.632084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:06.632162 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:06.632401 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:06.632403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:06.632404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:06.632643 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:06.632747 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:06.633168 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 16:38:06.633290 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":75,"threads_started":1}
I20260928 16:38:06.634395 12663 tablet_copy_service.cc:140] P f4821abf0a5b4e1c90d7ebcfb83e51d7: Received BeginTabletCopySession request for tablet 8c46e34d1037476691afe24be2674731 from peer c2ff3fe984f54f04a56ed3377d88fdda ({username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:52371)
I20260928 16:38:06.634546 12663 tablet_copy_service.cc:161] P f4821abf0a5b4e1c90d7ebcfb83e51d7: Beginning new tablet copy session on tablet 8c46e34d1037476691afe24be2674731 from peer c2ff3fe984f54f04a56ed3377d88fdda at {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:52371: session id = c2ff3fe984f54f04a56ed3377d88fdda-8c46e34d1037476691afe24be2674731
I20260928 16:38:06.635625 12663 tablet_copy_source_session.cc:205] T 8c46e34d1037476691afe24be2674731 P f4821abf0a5b4e1c90d7ebcfb83e51d7: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 16:38:06.639748 12850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c46e34d1037476691afe24be2674731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:06.645943 12850 tablet_copy_client.cc:791] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: tablet copy: Starting download of 0 data blocks...
I20260928 16:38:06.646128 12850 tablet_copy_client.cc:655] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: tablet copy: Starting download of 1 WAL segments...
I20260928 16:38:06.647779 12850 tablet_copy_client.cc:531] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 16:38:06.649554 12850 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Bootstrap starting.
I20260928 16:38:06.654568 12850 log.cc:824] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:06.660764 12850 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 16:38:06.661123 12850 tablet_bootstrap.cc:492] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Bootstrap complete.
I20260928 16:38:06.661427 12850 ts_tablet_manager.cc:1397] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.004s
I20260928 16:38:06.663661 12850 raft_consensus.cc:348] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:38:06.663901 12850 raft_consensus.cc:729] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ff3fe984f54f04a56ed3377d88fdda, State: Initialized, Role: FOLLOWER
I20260928 16:38:06.664325 12850 consensus_queue.cc:260] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c2ff3fe984f54f04a56ed3377d88fdda" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 35433 } } peers { permanent_uuid: "f4821abf0a5b4e1c90d7ebcfb83e51d7" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41341 } } peers { permanent_uuid: "1461e8d6cf9b4e98bd773aff186b5a01" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 37977 } }
I20260928 16:38:06.665205 12850 ts_tablet_manager.cc:1428] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260928 16:38:06.665433 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:06.665201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:58603)
0928 16:38:06.665312 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:06.665370 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 16:38:06.665877 12663 tablet_copy_service.cc:342] P f4821abf0a5b4e1c90d7ebcfb83e51d7: Request end of tablet copy session c2ff3fe984f54f04a56ed3377d88fdda-8c46e34d1037476691afe24be2674731 received from {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:52371
I20260928 16:38:06.665992 12663 tablet_copy_service.cc:433] P f4821abf0a5b4e1c90d7ebcfb83e51d7: ending tablet copy session c2ff3fe984f54f04a56ed3377d88fdda-8c46e34d1037476691afe24be2674731 on tablet 8c46e34d1037476691afe24be2674731 with peer c2ff3fe984f54f04a56ed3377d88fdda
I20260928 16:38:06.666179 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:06.665907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:52633)
0928 16:38:06.666063 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:06.666115 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 16:38:07.198002 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:07.197579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:57221)
0928 16:38:07.197840 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:07.197895 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
I20260928 16:38:07.198000 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:07.197590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:43959)
0928 16:38:07.197859 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:07.197911 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
I20260928 16:38:07.227943 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:07.227640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:54057)
0928 16:38:07.227772 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:07.227836 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 16:38:07.228231 12781 raft_consensus.cc:1203] T 8c46e34d1037476691afe24be2674731 P c2ff3fe984f54f04a56ed3377d88fdda [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260928 16:38:07.229387 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:07.229022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:49761)
0928 16:38:07.229204 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:07.229283 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260928 16:38:08.199375 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:08.198899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:46265)
0928 16:38:08.199202 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:08.199270 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":102,"threads_started":1}
I20260928 16:38:08.229228 12861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:08.228764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:54845)
0928 16:38:08.229066 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:08.229131 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":78,"threads_started":1}
I20260928 16:38:08.230436 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:08.230020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:52465)
0928 16:38:08.230307 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:08.230357 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":76,"threads_started":1}
I20260928 16:38:09.200791 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:09.200353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:55503)
0928 16:38:09.200631 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:09.200690 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":95,"threads_started":1}
I20260928 16:38:09.230367 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:09.229936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:39285)
0928 16:38:09.230218 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:09.230272 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":123,"threads_started":1}
I20260928 16:38:09.231477 12865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:09.231087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:44025)
0928 16:38:09.231324 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:09.231371 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
I20260928 16:38:10.201929 12867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:10.201500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:56737)
0928 16:38:10.201776 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:10.201826 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":120,"threads_started":1}
I20260928 16:38:10.231767 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:10.231282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:33495)
0928 16:38:10.231588 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:10.231644 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260928 16:38:10.232764 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:10.232346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:58799)
0928 16:38:10.232585 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:10.232638 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":59,"threads_started":1}
I20260928 16:38:10.232957 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:10.232658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:58971)
0928 16:38:10.232809 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:10.232867 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 16:38:11.203156 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:11.202710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:47499)
0928 16:38:11.202993 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:11.203060 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260928 16:38:11.233978 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:11.233560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:58979)
0928 16:38:11.233821 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:11.233870 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":108,"threads_started":1}
I20260928 16:38:11.234118 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:11.233666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:35529)
0928 16:38:11.233924 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:11.233976 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":94,"threads_started":1}
I20260928 16:38:12.204806 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:12.204197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:51233)
0928 16:38:12.204572 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:12.204646 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":112,"threads_started":1}
I20260928 16:38:12.235894 12875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:12.235135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:47991)
0928 16:38:12.235601 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:12.235698 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":184,"threads_started":1}
I20260928 16:38:12.236049 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:12.235162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:48631)
0928 16:38:12.235744 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:12.235883 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":247,"threads_started":1}
I20260928 16:38:13.206157 12879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:13.205663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:38931)
0928 16:38:13.206005 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:13.206062 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":90,"threads_started":1}
I20260928 16:38:13.237399 12881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:13.236945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:49227)
0928 16:38:13.237240 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:13.237300 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":98,"threads_started":1}
I20260928 16:38:13.237444 12880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:13.236945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:54759)
0928 16:38:13.237307 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:13.237358 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":99,"threads_started":1}
I20260928 16:38:14.207561 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:14.207138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:54123)
0928 16:38:14.207401 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:14.207456 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
I20260928 16:38:14.238663 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:14.238192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:42801)
0928 16:38:14.238473 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:14.238526 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
I20260928 16:38:14.238682 12883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:14.238192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:41813)
0928 16:38:14.238495 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:14.238547 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":75,"threads_started":1}
I20260928 16:38:15.208658 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:15.208264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:56137)
0928 16:38:15.208506 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:15.208564 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
I20260928 16:38:15.239992 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:15.239555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:41483)
0928 16:38:15.239827 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:15.239884 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260928 16:38:15.240013 12886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:15.239566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:60357)
0928 16:38:15.239870 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:15.239919 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":78,"threads_started":1}
I20260928 16:38:16.209785 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:16.209358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:33095)
0928 16:38:16.209634 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:16.209688 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":96,"threads_started":1}
I20260928 16:38:16.241230 12889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:16.240732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:45187)
0928 16:38:16.241057 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:16.241119 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":87,"threads_started":1}
I20260928 16:38:16.241230 12890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:16.240732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:39611)
0928 16:38:16.241057 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:16.241119 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":89,"threads_started":1}
I20260928 16:38:17.210897 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:17.210501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:48991)
0928 16:38:17.210748 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:17.210802 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260928 16:38:17.242479 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:17.242041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:37735)
0928 16:38:17.242293 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:17.242352 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260928 16:38:17.242515 12892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:17.241998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:53819)
0928 16:38:17.242377 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:17.242429 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":125,"threads_started":1}
I20260928 16:38:18.212025 12894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:18.211599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:49365)
0928 16:38:18.211862 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:18.211925 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":90,"threads_started":1}
I20260928 16:38:18.243882 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:18.243295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:58589)
0928 16:38:18.243690 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:18.243761 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":125,"threads_started":1}
I20260928 16:38:18.243882 12896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:18.243384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:51129)
0928 16:38:18.243708 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:18.243769 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":72,"threads_started":1}
I20260928 16:38:19.213469 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:19.213056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.193:35991)
0928 16:38:19.213308 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:19.213373 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260928 16:38:19.245252 12898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:19.244776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.195:53129)
0928 16:38:19.245056 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:19.245129 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":138,"threads_started":1}
I20260928 16:38:19.245349 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:19.244890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38467 (local address 127.27.122.194:51429)
0928 16:38:19.245199 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:19.245253 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.254:38467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":79,"threads_started":1}
I20260928 16:38:19.438212 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 12381
I20260928 16:38:19.444447 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 12554
I20260928 16:38:19.451383 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 12694
2026-09-28T16:38:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59131 ms)
[----------] 30 tests from SecurityITest (243805 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:19 dist-test-slave-g9sh krb5kdc[12903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:19 dist-test-slave-g9sh krb5kdc[12903](info): set up 2 sockets
Sep 28 16:38:19 dist-test-slave-g9sh krb5kdc[12903](info): commencing operation
krb5kdc: starting...
W20260928 16:38:20.499462 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:20 dist-test-slave-g9sh krb5kdc[12903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:38:20.620060 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:41105
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:41105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:20.728328 12916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:20.728601 12916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:20.728669 12916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:20.732566 12916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:20.732648 12916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:20.732686 12916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:20.732739 12916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:20.732777 12916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:20.737830 12916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:41105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:41105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:20.738816 12916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:20.739796 12916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:38:20.748363 12916 server_base.cc:1034] running on GCE node
W20260928 16:38:20.748413 12924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:20.748420 12921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:20.748409 12922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:20.749265 12916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:20.750311 12916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:20.751497 12916 hybrid_clock.cc:648] HybridClock initialized: now 1790613500751476 us; error 29 us; skew 500 ppm
Sep 28 16:38:20 dist-test-slave-g9sh krb5kdc[12903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613500, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:20.754173 12916 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:38:20.754944 12916 webserver.cc:460] Webserver started at http://127.27.122.254:45565/ using document root <none> and password file <none>
I20260928 16:38:20.755434 12916 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:20.755489 12916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:20.755646 12916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:20.757592 12916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "b4460c836dbb4e749dc15d4c6da1648c"
format_stamp: "Formatted at 2026-09-28 16:38:20 on dist-test-slave-g9sh"
server_key: "a7c52051cb6b73f84ce2a597d9377869"
server_key_iv: "d25039a6a304a28b4ab018b9ed870178"
server_key_version: "encryptionkey@0"
I20260928 16:38:20.758091 12916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "b4460c836dbb4e749dc15d4c6da1648c"
format_stamp: "Formatted at 2026-09-28 16:38:20 on dist-test-slave-g9sh"
server_key: "a7c52051cb6b73f84ce2a597d9377869"
server_key_iv: "d25039a6a304a28b4ab018b9ed870178"
server_key_version: "encryptionkey@0"
I20260928 16:38:20.761583 12916 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:38:20.763734 12931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:20.764845 12916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:38:20.765007 12916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "b4460c836dbb4e749dc15d4c6da1648c"
format_stamp: "Formatted at 2026-09-28 16:38:20 on dist-test-slave-g9sh"
server_key: "a7c52051cb6b73f84ce2a597d9377869"
server_key_iv: "d25039a6a304a28b4ab018b9ed870178"
server_key_version: "encryptionkey@0"
I20260928 16:38:20.765094 12916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:20.775916 12916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:20.778594 12916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:20.778730 12916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:20.785506 12916 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:41105
I20260928 16:38:20.785524 12986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:41105 every 8 connection(s)
I20260928 16:38:20.786377 12916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:20.789037 12987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:20.794451 12987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c: Bootstrap starting.
I20260928 16:38:20.795521 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 12916
I20260928 16:38:20.795663 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:20.795972 28139 external_mini_cluster.cc:1428] Setting key 8def0a7be14159d266c88fbdf31d5243
I20260928 16:38:20.797271 12987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:20.798014 12987 log.cc:824] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:20.799860 12987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c: No bootstrap required, opened a new log
I20260928 16:38:20.803224 12987 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4460c836dbb4e749dc15d4c6da1648c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41105 } }
I20260928 16:38:20.803448 12987 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:20.803526 12987 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4460c836dbb4e749dc15d4c6da1648c, State: Initialized, Role: FOLLOWER
I20260928 16:38:20.803988 12987 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4460c836dbb4e749dc15d4c6da1648c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41105 } }
I20260928 16:38:20.804176 12987 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:20.804251 12987 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:20.804349 12987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [term 0 FOLLOWER]: Advancing to term 1
Sep 28 16:38:20 dist-test-slave-g9sh krb5kdc[12903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:20.805428 12987 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4460c836dbb4e749dc15d4c6da1648c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41105 } }
I20260928 16:38:20.805796 12987 leader_election.cc:302] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4460c836dbb4e749dc15d4c6da1648c; no voters:
I20260928 16:38:20.806111 12987 leader_election.cc:288] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:20.806201 12992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:20.806401 12992 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [term 1 LEADER]: Becoming Leader. State: Replica: b4460c836dbb4e749dc15d4c6da1648c, State: Running, Role: LEADER
I20260928 16:38:20.806692 12992 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4460c836dbb4e749dc15d4c6da1648c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41105 } }
I20260928 16:38:20.807279 12987 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:20.807878 12994 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4460c836dbb4e749dc15d4c6da1648c. Latest consensus state: current_term: 1 leader_uuid: "b4460c836dbb4e749dc15d4c6da1648c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4460c836dbb4e749dc15d4c6da1648c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41105 } } }
I20260928 16:38:20.807989 12994 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:20.808362 12999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:20.810647 12993 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4460c836dbb4e749dc15d4c6da1648c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4460c836dbb4e749dc15d4c6da1648c" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41105 } } }
I20260928 16:38:20.810760 12993 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:20.811369 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:20.797103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.27.122.254:41105)
0928 16:38:20.797542 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:20.797557 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:20.797590 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:38:20.798277 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:20.798296 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:20.798710 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:20.799063 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:20.799870 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:20.800881 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:20.801795 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:20.802275 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:20.802434 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:20.805230 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:20.805235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:20.805247 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:20.805272 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:20.807602 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:20.808256 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:20.808265 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:20.808272 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:20.808413 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:20.808787 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:20.808793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:20.808798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:20.809398 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:20.809618 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:20.809924 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 16:38:20.810232 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":131,"threads_started":1}
I20260928 16:38:20.811872 12999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:20.817236 12999 catalog_manager.cc:1347] Generated new cluster ID: 4631deb91a444a2cb5c2329f9396fee1
I20260928 16:38:20.817302 12999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:20.851999 12999 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:20.853034 12999 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:20.858615 12999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c: Generated new TSK 0
I20260928 16:38:20.859267 12999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:38:20.894341 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41105
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:21.005983 13014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:21.006203 13014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:21.006242 13014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:21.010406 13014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:21.010452 13014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:21.010552 13014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:38:21.015514 13014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:21.016426 13014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:21.017410 13014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:21.026645 13019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:21.026665 13020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:21.026729 13014 server_base.cc:1034] running on GCE node
W20260928 16:38:21.026680 13022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:21.027245 13014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:21.027863 13014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:21.029030 13014 hybrid_clock.cc:648] HybridClock initialized: now 1790613501029012 us; error 28 us; skew 500 ppm
Sep 28 16:38:21 dist-test-slave-g9sh krb5kdc[12903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613501, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:21.031749 13014 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:38:21.032559 13014 webserver.cc:460] Webserver started at http://127.27.122.193:36929/ using document root <none> and password file <none>
I20260928 16:38:21.033130 13014 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:21.033212 13014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:21.033424 13014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:21.035311 13014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "08b377e3cc2b4a81a13d0d3f48c569d6"
format_stamp: "Formatted at 2026-09-28 16:38:21 on dist-test-slave-g9sh"
server_key: "0652247ae1c90058f00ef7e146adc3d7"
server_key_iv: "269882d198a2bd45138fee55fe74ecf0"
server_key_version: "encryptionkey@0"
I20260928 16:38:21.035861 13014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "08b377e3cc2b4a81a13d0d3f48c569d6"
format_stamp: "Formatted at 2026-09-28 16:38:21 on dist-test-slave-g9sh"
server_key: "0652247ae1c90058f00ef7e146adc3d7"
server_key_iv: "269882d198a2bd45138fee55fe74ecf0"
server_key_version: "encryptionkey@0"
I20260928 16:38:21.039304 13014 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 16:38:21.041644 13030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:21.042836 13014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:38:21.042975 13014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "08b377e3cc2b4a81a13d0d3f48c569d6"
format_stamp: "Formatted at 2026-09-28 16:38:21 on dist-test-slave-g9sh"
server_key: "0652247ae1c90058f00ef7e146adc3d7"
server_key_iv: "269882d198a2bd45138fee55fe74ecf0"
server_key_version: "encryptionkey@0"
I20260928 16:38:21.043081 13014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:21.051962 13014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:21.054554 13014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:21.054741 13014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:21.055263 13014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:21.056034 13014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:21.056100 13014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:21.056167 13014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:21.056219 13014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:21.064987 13014 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:39671
I20260928 16:38:21.065016 13145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:39671 every 8 connection(s)
I20260928 16:38:21.065796 13014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:38:21.069957 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 13014
I20260928 16:38:21.070214 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:38:21.070494 28139 external_mini_cluster.cc:1428] Setting key 2c780e50cbe32a72da24ddcb6c87e9fd
Sep 28 16:38:21 dist-test-slave-g9sh krb5kdc[12903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613501, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:21.079530 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.067516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:36381 (local address 127.27.122.254:41105)
0928 16:38:21.067697 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:21.067702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:21.068480 (+ 778us) server_negotiation.cc:407] Connection header received
0928 16:38:21.069883 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:21.069886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:21.069946 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:21.070051 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:21.071310 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.071948 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.073132 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.073436 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.073567 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.075972 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:21.075976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:21.075978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:21.076004 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:21.077523 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:21.078015 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:21.078022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:21.078025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:21.078115 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:21.078445 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:21.078450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:21.078452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:21.078635 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:21.078743 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:21.079222 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 16:38:21.079359 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:38:21.080404 13148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.067919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41105 (local address 127.27.122.193:36381)
0928 16:38:21.068344 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:21.068383 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:21.069672 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:21.070225 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:21.070238 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:21.070633 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:21.071161 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.071184 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.072070 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.072073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.072996 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.073012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.073612 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.073616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.073766 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.074363 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:21.074378 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:21.075806 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:21.077641 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:21.077647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:21.077660 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:21.077911 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:21.078208 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:21.078211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:21.078213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:21.078351 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:21.078741 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:21.078748 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:21.078986 (+ 238us) client_negotiation.cc:769] Sending connection context
0928 16:38:21.079177 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 16:38:21.079502 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260928 16:38:21.081687 13146 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41105
I20260928 16:38:21.081905 13146 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
I20260928 16:38:21.082427 13146 heartbeater.cc:511] Master 127.27.122.254:41105 requested a full tablet report, sending...
I20260928 16:38:21.084076 12950 ts_manager.cc:194] Registered new tserver with Master: 08b377e3cc2b4a81a13d0d3f48c569d6 (127.27.122.193:39671)
I20260928 16:38:21.085230 12950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:36381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:38:21.097957 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41105
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:21.206983 13151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:21.207243 13151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:21.207324 13151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:21.211256 13151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:21.211333 13151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:21.211416 13151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:38:21.216430 13151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:21.217393 13151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:21.218348 13151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:21.227705 13159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:21.227718 13157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:21.227710 13156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:21.227751 13151 server_base.cc:1034] running on GCE node
I20260928 16:38:21.228242 13151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:21.228852 13151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:21.230052 13151 hybrid_clock.cc:648] HybridClock initialized: now 1790613501230026 us; error 36 us; skew 500 ppm
Sep 28 16:38:21 dist-test-slave-g9sh krb5kdc[12903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613501, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:21.232746 13151 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:38:21.233603 13151 webserver.cc:460] Webserver started at http://127.27.122.194:34073/ using document root <none> and password file <none>
I20260928 16:38:21.234138 13151 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:21.234222 13151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:21.234436 13151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:21.236436 13151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "4889805ad4644e51a4b6e01b08a6477f"
format_stamp: "Formatted at 2026-09-28 16:38:21 on dist-test-slave-g9sh"
server_key: "e05306ccc8f3cd4fd629e0431c338705"
server_key_iv: "a1d3756c231482387efa201a7bbe03b0"
server_key_version: "encryptionkey@0"
I20260928 16:38:21.237049 13151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "4889805ad4644e51a4b6e01b08a6477f"
format_stamp: "Formatted at 2026-09-28 16:38:21 on dist-test-slave-g9sh"
server_key: "e05306ccc8f3cd4fd629e0431c338705"
server_key_iv: "a1d3756c231482387efa201a7bbe03b0"
server_key_version: "encryptionkey@0"
I20260928 16:38:21.240638 13151 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 16:38:21.243031 13166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:21.244266 13151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:38:21.244400 13151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "4889805ad4644e51a4b6e01b08a6477f"
format_stamp: "Formatted at 2026-09-28 16:38:21 on dist-test-slave-g9sh"
server_key: "e05306ccc8f3cd4fd629e0431c338705"
server_key_iv: "a1d3756c231482387efa201a7bbe03b0"
server_key_version: "encryptionkey@0"
I20260928 16:38:21.244526 13151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:21.262866 13151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:21.265661 13151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:21.265844 13151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:21.266374 13151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:21.267144 13151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:21.267212 13151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:21.267280 13151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:21.267336 13151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:21.275766 13151 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:45887
I20260928 16:38:21.275790 13282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:45887 every 8 connection(s)
I20260928 16:38:21.276582 13151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:38:21.283876 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 13151
I20260928 16:38:21.284027 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:38:21.284379 28139 external_mini_cluster.cc:1428] Setting key ca792ce6e2d9e765fc03ca693619ad2f
Sep 28 16:38:21 dist-test-slave-g9sh krb5kdc[12903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613501, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:21.290301 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.278260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:50237 (local address 127.27.122.254:41105)
0928 16:38:21.278383 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:21.278386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:21.279115 (+ 729us) server_negotiation.cc:407] Connection header received
0928 16:38:21.280356 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:21.280358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:21.280407 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:21.280478 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:21.281641 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.282170 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.283283 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.283507 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.283597 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.286248 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:21.286252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:21.286255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:21.286281 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:21.287932 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:21.288553 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:21.288556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:21.288557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:21.288656 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:21.289063 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:21.289066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:21.289068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:21.289233 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:21.289312 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:21.289908 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 16:38:21.290051 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:38:21.290841 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.278550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41105 (local address 127.27.122.194:50237)
0928 16:38:21.278969 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:21.279012 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:21.280172 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:21.280647 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:21.280655 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:21.281047 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:21.281511 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.281524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.282275 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.282277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.283150 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.283160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.283665 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.283672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.283886 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.284452 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:21.284474 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:21.286099 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:21.288062 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:21.288070 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:21.288084 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:21.288435 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:21.288744 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:21.288748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:21.288752 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:21.288957 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:21.289348 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:21.289353 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:21.289649 (+ 296us) client_negotiation.cc:769] Sending connection context
0928 16:38:21.289799 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 16:38:21.290051 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
I20260928 16:38:21.291898 13283 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41105
I20260928 16:38:21.292178 13283 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:21.292771 13283 heartbeater.cc:511] Master 127.27.122.254:41105 requested a full tablet report, sending...
I20260928 16:38:21.293900 12950 ts_manager.cc:194] Registered new tserver with Master: 4889805ad4644e51a4b6e01b08a6477f (127.27.122.194:45887)
I20260928 16:38:21.294464 12950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:50237
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:38:21.311976 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41105
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:21.422456 13288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:21.422672 13288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:21.422709 13288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:21.426797 13288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:21.426851 13288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:21.426906 13288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:38:21.431859 13288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:41105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:21.432706 13288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:21.433652 13288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:21.442941 13294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:21.442888 13295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:21.442888 13297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:21.443145 13288 server_base.cc:1034] running on GCE node
I20260928 16:38:21.443492 13288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:21.444110 13288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:21.445283 13288 hybrid_clock.cc:648] HybridClock initialized: now 1790613501445263 us; error 34 us; skew 500 ppm
Sep 28 16:38:21 dist-test-slave-g9sh krb5kdc[12903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613501, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:21.447999 13288 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:38:21.448833 13288 webserver.cc:460] Webserver started at http://127.27.122.195:36599/ using document root <none> and password file <none>
I20260928 16:38:21.449362 13288 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:21.449422 13288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:21.449654 13288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:21.451503 13288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "6b0a86114f234e7bad2db7d66caac7ce"
format_stamp: "Formatted at 2026-09-28 16:38:21 on dist-test-slave-g9sh"
server_key: "42b022c7388aafd582a6959faf318007"
server_key_iv: "7f4ed27f47eb74894a7590512bd20ecc"
server_key_version: "encryptionkey@0"
I20260928 16:38:21.452028 13288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "6b0a86114f234e7bad2db7d66caac7ce"
format_stamp: "Formatted at 2026-09-28 16:38:21 on dist-test-slave-g9sh"
server_key: "42b022c7388aafd582a6959faf318007"
server_key_iv: "7f4ed27f47eb74894a7590512bd20ecc"
server_key_version: "encryptionkey@0"
I20260928 16:38:21.457017 13288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 16:38:21.459483 13304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:21.460618 13288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:38:21.460731 13288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "6b0a86114f234e7bad2db7d66caac7ce"
format_stamp: "Formatted at 2026-09-28 16:38:21 on dist-test-slave-g9sh"
server_key: "42b022c7388aafd582a6959faf318007"
server_key_iv: "7f4ed27f47eb74894a7590512bd20ecc"
server_key_version: "encryptionkey@0"
I20260928 16:38:21.460837 13288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:21.479353 13288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:21.481976 13288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:21.482154 13288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:21.482688 13288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:21.483460 13288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:21.483505 13288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:21.483573 13288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:21.483610 13288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:21.492424 13288 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:38821
I20260928 16:38:21.492519 13421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:38821 every 8 connection(s)
I20260928 16:38:21.493346 13288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:38:21.497394 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 13288
I20260928 16:38:21.497548 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:38:21.497893 28139 external_mini_cluster.cc:1428] Setting key 689a08ed12a085ffa88cbfb5851baa2d
Sep 28 16:38:21 dist-test-slave-g9sh krb5kdc[12903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613501, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:21.506834 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.495128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:33563 (local address 127.27.122.254:41105)
0928 16:38:21.495283 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:21.495288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:21.496150 (+ 862us) server_negotiation.cc:407] Connection header received
0928 16:38:21.497529 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:21.497533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:21.497596 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:21.497690 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:21.499126 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.499672 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.500811 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.501101 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.501209 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.503591 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:21.503595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:21.503603 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:21.503627 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:21.504856 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:21.505397 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:21.505401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:21.505403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:21.505493 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:21.505813 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:21.505817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:21.505819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:21.506000 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:21.506101 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:21.506563 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 16:38:21.506678 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:38:21.507680 13424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.495533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:41105 (local address 127.27.122.195:33563)
0928 16:38:21.495998 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:21.496042 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:21.497294 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:21.498006 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:21.498015 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:21.498414 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:21.498939 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.498959 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.499799 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.499803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.500685 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.500695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.501247 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.501251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.501416 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.501969 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:21.501985 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:21.503436 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:21.505027 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:21.505033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:21.505046 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:21.505297 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:21.505582 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:21.505586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:21.505588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:21.505720 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:21.506111 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:21.506119 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:21.506344 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 16:38:21.506521 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 16:38:21.506851 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":103,"threads_started":1}
I20260928 16:38:21.508761 13422 heartbeater.cc:347] Connected to a master server at 127.27.122.254:41105
I20260928 16:38:21.509014 13422 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:21.509480 13422 heartbeater.cc:511] Master 127.27.122.254:41105 requested a full tablet report, sending...
I20260928 16:38:21.510491 12950 ts_manager.cc:194] Registered new tserver with Master: 6b0a86114f234e7bad2db7d66caac7ce (127.27.122.195:38821)
I20260928 16:38:21.511065 12950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:33563
I20260928 16:38:21.511596 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:38:21.520694 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.513604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35376 (local address 127.27.122.254:41105)
0928 16:38:21.513725 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:21.513729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:21.513901 (+ 172us) server_negotiation.cc:407] Connection header received
0928 16:38:21.513985 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:21.513988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:21.514023 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:21.514082 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:21.514700 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.515168 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.515957 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.516173 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.516264 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.517575 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:21.517578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:21.517580 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:21.517602 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:21.518833 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:21.519252 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:21.519254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:21.519256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:21.519327 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:21.519628 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:21.519631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:21.519632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:21.519832 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:21.519946 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:21.520377 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 16:38:21.520472 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 16:38:21.523643 12949 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:38:21.526759 12949 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:38:21.541185 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.535162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.27.122.195:38821)
0928 16:38:21.535782 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:21.535796 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:21.535822 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:38:21.535904 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:21.535910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:21.536151 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:21.536689 (+ 538us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:21.536851 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.538066 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.539749 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.540446 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.540564 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.540693 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:21.540927 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 16:38:21.541031 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":166,"threads_started":1}
I20260928 16:38:21.541236 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.535195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:39671 (local address 127.0.0.1:55162)
0928 16:38:21.535931 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:21.535944 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:21.536026 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:21.536843 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:21.536846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:21.536863 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:21.537011 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.537018 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.538381 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.538385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.539588 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.539600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.540777 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.540781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.540871 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.540920 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 16:38:21.541009 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 16:38:21.541093 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":42,"thread_start_us":101,"threads_started":1}
I20260928 16:38:21.541325 13433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.535008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:38821 (local address 127.0.0.1:56034)
0928 16:38:21.535350 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:21.535402 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:21.535535 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:21.536333 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:21.536342 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:21.536374 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:21.536532 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.536540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.538199 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.538202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.539621 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.539631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.540579 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.540583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.540696 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.540733 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:38:21.540868 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 16:38:21.541014 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":81,"thread_start_us":90,"threads_started":1}
I20260928 16:38:21.541687 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.536292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:45887 (local address 127.0.0.1:32852)
0928 16:38:21.537053 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:21.537067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:21.537151 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:21.537543 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:21.537545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:21.537562 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:21.537664 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.537674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.538899 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.538902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.540049 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.540058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.541270 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.541274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.541372 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.541391 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 16:38:21.541475 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 16:38:21.541551 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260928 16:38:21.541742 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.536454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.27.122.194:45887)
0928 16:38:21.536883 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:21.536918 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:21.537238 (+ 320us) server_negotiation.cc:407] Connection header received
0928 16:38:21.537303 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:21.537306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:21.537425 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:21.537563 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:21.537790 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.538757 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.540168 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.540997 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.541101 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.541208 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:21.541514 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 16:38:21.541606 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":142,"threads_started":1}
I20260928 16:38:21.542047 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.535813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.27.122.193:39671)
0928 16:38:21.536176 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:21.536185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:21.536204 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:38:21.536296 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:21.536301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:21.536516 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:21.536677 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:21.537146 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.538252 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.539728 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.540468 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.540602 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.540730 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:21.541407 (+ 677us) server_negotiation.cc:299] Negotiation successful
0928 16:38:21.541591 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":172,"threads_started":1}
I20260928 16:38:21.544487 13354 tablet_service.cc:1467] Processing CreateTablet for tablet a85abeea8b294e7594e7112eb45a9557 (DEFAULT_TABLE table=test-table [id=de26bcd29b57453fb8666834572586da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:21.544483 13078 tablet_service.cc:1467] Processing CreateTablet for tablet a85abeea8b294e7594e7112eb45a9557 (DEFAULT_TABLE table=test-table [id=de26bcd29b57453fb8666834572586da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:21.545511 13078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85abeea8b294e7594e7112eb45a9557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:21.545518 13354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85abeea8b294e7594e7112eb45a9557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:21.546761 13215 tablet_service.cc:1467] Processing CreateTablet for tablet a85abeea8b294e7594e7112eb45a9557 (DEFAULT_TABLE table=test-table [id=de26bcd29b57453fb8666834572586da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:21.547672 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85abeea8b294e7594e7112eb45a9557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:21.551601 13439 tablet_bootstrap.cc:492] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6: Bootstrap starting.
I20260928 16:38:21.553289 13440 tablet_bootstrap.cc:492] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce: Bootstrap starting.
I20260928 16:38:21.553891 13439 tablet_bootstrap.cc:654] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:21.554528 13441 tablet_bootstrap.cc:492] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f: Bootstrap starting.
I20260928 16:38:21.555291 13440 tablet_bootstrap.cc:654] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:21.555464 13439 log.cc:824] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:21.556464 13440 log.cc:824] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:21.556530 13441 tablet_bootstrap.cc:654] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:21.557673 13439 tablet_bootstrap.cc:492] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6: No bootstrap required, opened a new log
I20260928 16:38:21.557866 13439 ts_tablet_manager.cc:1397] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:38:21.557912 13441 log.cc:824] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:21.558255 13440 tablet_bootstrap.cc:492] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce: No bootstrap required, opened a new log
I20260928 16:38:21.558429 13440 ts_tablet_manager.cc:1397] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:38:21.559700 13441 tablet_bootstrap.cc:492] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f: No bootstrap required, opened a new log
I20260928 16:38:21.559880 13441 ts_tablet_manager.cc:1397] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:38:21.560981 13439 raft_consensus.cc:348] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b0a86114f234e7bad2db7d66caac7ce" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38821 } } peers { permanent_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39671 } } peers { permanent_uuid: "4889805ad4644e51a4b6e01b08a6477f" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45887 } }
I20260928 16:38:21.561213 13439 raft_consensus.cc:374] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:21.561267 13439 raft_consensus.cc:729] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b377e3cc2b4a81a13d0d3f48c569d6, State: Initialized, Role: FOLLOWER
I20260928 16:38:21.561695 13439 consensus_queue.cc:260] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b0a86114f234e7bad2db7d66caac7ce" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38821 } } peers { permanent_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39671 } } peers { permanent_uuid: "4889805ad4644e51a4b6e01b08a6477f" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45887 } }
I20260928 16:38:21.562323 13146 heartbeater.cc:503] Master 127.27.122.254:41105 was elected leader, sending a full tablet report...
I20260928 16:38:21.562188 13440 raft_consensus.cc:348] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b0a86114f234e7bad2db7d66caac7ce" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38821 } } peers { permanent_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39671 } } peers { permanent_uuid: "4889805ad4644e51a4b6e01b08a6477f" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45887 } }
I20260928 16:38:21.562414 13440 raft_consensus.cc:374] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:21.562469 13440 raft_consensus.cc:729] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0a86114f234e7bad2db7d66caac7ce, State: Initialized, Role: FOLLOWER
I20260928 16:38:21.562668 13439 ts_tablet_manager.cc:1428] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:38:21.562898 13440 consensus_queue.cc:260] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b0a86114f234e7bad2db7d66caac7ce" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38821 } } peers { permanent_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39671 } } peers { permanent_uuid: "4889805ad4644e51a4b6e01b08a6477f" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45887 } }
I20260928 16:38:21.563546 13422 heartbeater.cc:503] Master 127.27.122.254:41105 was elected leader, sending a full tablet report...
I20260928 16:38:21.563920 13441 raft_consensus.cc:348] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b0a86114f234e7bad2db7d66caac7ce" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38821 } } peers { permanent_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39671 } } peers { permanent_uuid: "4889805ad4644e51a4b6e01b08a6477f" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45887 } }
I20260928 16:38:21.564208 13441 raft_consensus.cc:374] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:21.564265 13441 raft_consensus.cc:729] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4889805ad4644e51a4b6e01b08a6477f, State: Initialized, Role: FOLLOWER
I20260928 16:38:21.564687 13441 consensus_queue.cc:260] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b0a86114f234e7bad2db7d66caac7ce" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38821 } } peers { permanent_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39671 } } peers { permanent_uuid: "4889805ad4644e51a4b6e01b08a6477f" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45887 } }
I20260928 16:38:21.565347 13440 ts_tablet_manager.cc:1428] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:38:21.565709 13441 ts_tablet_manager.cc:1428] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 16:38:21.565853 13283 heartbeater.cc:503] Master 127.27.122.254:41105 was elected leader, sending a full tablet report...
W20260928 16:38:21.566815 13147 tablet.cc:2369] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:38:21.719266 13445 raft_consensus.cc:482] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:38:21.719488 13445 raft_consensus.cc:504] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b0a86114f234e7bad2db7d66caac7ce" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38821 } } peers { permanent_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39671 } } peers { permanent_uuid: "4889805ad4644e51a4b6e01b08a6477f" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45887 } }
I20260928 16:38:21.720639 13445 leader_election.cc:288] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b0a86114f234e7bad2db7d66caac7ce (127.27.122.195:38821), 4889805ad4644e51a4b6e01b08a6477f (127.27.122.194:45887)
I20260928 16:38:21.724638 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.721013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:47751 (local address 127.27.122.194:45887)
0928 16:38:21.721165 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:21.721171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:21.721498 (+ 327us) server_negotiation.cc:407] Connection header received
0928 16:38:21.721620 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:21.721622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:21.721674 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:21.721759 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:21.722059 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.722563 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.723665 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.724075 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.724189 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.724247 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:21.724438 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 16:38:21.724511 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:38:21.724663 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.721012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:59745 (local address 127.27.122.195:38821)
0928 16:38:21.721228 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:21.721234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:21.721262 (+ 28us) server_negotiation.cc:407] Connection header received
0928 16:38:21.721443 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:21.721449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:21.721511 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:21.721599 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:21.721933 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.722499 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.723705 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.724101 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.724231 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.724285 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:21.724478 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 16:38:21.724552 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260928 16:38:21.724653 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.720887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:45887 (local address 127.27.122.193:47751)
0928 16:38:21.721384 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:21.721397 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:21.721501 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:21.721779 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:21.721781 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:21.721796 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:21.721940 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.721946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.722679 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.722681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.723532 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.723540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.724199 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.724203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.724296 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.724326 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:38:21.724392 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 16:38:21.724481 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":25,"thread_start_us":75,"threads_started":1}
I20260928 16:38:21.724669 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.720861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:38821 (local address 127.27.122.193:59745)
0928 16:38:21.721182 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:21.721199 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:21.721307 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:21.721625 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:21.721627 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:21.721645 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:21.721809 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.721817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.722628 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.722634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.723541 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:21.723550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.724239 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.724243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:21.724337 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.724354 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 16:38:21.724444 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 16:38:21.724511 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":27,"thread_start_us":315,"threads_started":1}
I20260928 16:38:21.725368 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85abeea8b294e7594e7112eb45a9557" candidate_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4889805ad4644e51a4b6e01b08a6477f" is_pre_election: true
I20260928 16:38:21.725553 13374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85abeea8b294e7594e7112eb45a9557" candidate_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b0a86114f234e7bad2db7d66caac7ce" is_pre_election: true
I20260928 16:38:21.725701 13235 raft_consensus.cc:2454] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08b377e3cc2b4a81a13d0d3f48c569d6 in term 0.
I20260928 16:38:21.725816 13374 raft_consensus.cc:2454] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08b377e3cc2b4a81a13d0d3f48c569d6 in term 0.
I20260928 16:38:21.726220 13032 leader_election.cc:302] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b377e3cc2b4a81a13d0d3f48c569d6, 4889805ad4644e51a4b6e01b08a6477f; no voters:
I20260928 16:38:21.726511 13445 raft_consensus.cc:2781] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:38:21.726630 13445 raft_consensus.cc:482] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:38:21.726687 13445 raft_consensus.cc:3037] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:21.727905 13445 raft_consensus.cc:504] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b0a86114f234e7bad2db7d66caac7ce" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38821 } } peers { permanent_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39671 } } peers { permanent_uuid: "4889805ad4644e51a4b6e01b08a6477f" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45887 } }
I20260928 16:38:21.728269 13445 leader_election.cc:288] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [CANDIDATE]: Term 1 election: Requested vote from peers 6b0a86114f234e7bad2db7d66caac7ce (127.27.122.195:38821), 4889805ad4644e51a4b6e01b08a6477f (127.27.122.194:45887)
I20260928 16:38:21.728618 13374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85abeea8b294e7594e7112eb45a9557" candidate_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b0a86114f234e7bad2db7d66caac7ce"
I20260928 16:38:21.728672 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a85abeea8b294e7594e7112eb45a9557" candidate_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4889805ad4644e51a4b6e01b08a6477f"
I20260928 16:38:21.728760 13374 raft_consensus.cc:3037] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:21.728848 13235 raft_consensus.cc:3037] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:21.729849 13374 raft_consensus.cc:2454] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08b377e3cc2b4a81a13d0d3f48c569d6 in term 1.
I20260928 16:38:21.729988 13235 raft_consensus.cc:2454] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08b377e3cc2b4a81a13d0d3f48c569d6 in term 1.
I20260928 16:38:21.730176 13034 leader_election.cc:302] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b377e3cc2b4a81a13d0d3f48c569d6, 6b0a86114f234e7bad2db7d66caac7ce; no voters:
I20260928 16:38:21.730348 13445 raft_consensus.cc:2781] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:21.730578 13445 raft_consensus.cc:686] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [term 1 LEADER]: Becoming Leader. State: Replica: 08b377e3cc2b4a81a13d0d3f48c569d6, State: Running, Role: LEADER
I20260928 16:38:21.730888 13445 consensus_queue.cc:237] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b0a86114f234e7bad2db7d66caac7ce" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38821 } } peers { permanent_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39671 } } peers { permanent_uuid: "4889805ad4644e51a4b6e01b08a6477f" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45887 } }
I20260928 16:38:21.733916 12948 catalog_manager.cc:5495] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08b377e3cc2b4a81a13d0d3f48c569d6 (127.27.122.193). New cstate: current_term: 1 leader_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b0a86114f234e7bad2db7d66caac7ce" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08b377e3cc2b4a81a13d0d3f48c569d6" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4889805ad4644e51a4b6e01b08a6477f" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45887 } health_report { overall_health: UNKNOWN } } }
W20260928 16:38:21.744256 13423 tablet.cc:2369] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:38:21.777753 13284 tablet.cc:2369] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:38:21.782162 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.777706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.27.122.193:39671)
0928 16:38:21.777831 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:21.777837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:21.777908 (+ 71us) server_negotiation.cc:407] Connection header received
0928 16:38:21.778133 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:21.778136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:21.778209 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:21.778327 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:38:21.778651 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.779144 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.780614 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:21.780820 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:21.780889 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:21.781338 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:38:21.781757 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:38:21.781850 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:21.781966 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 16:38:21.782014 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:38:21.788128 13374 raft_consensus.cc:1261] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce [term 1 FOLLOWER]: Refusing update from remote peer 08b377e3cc2b4a81a13d0d3f48c569d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:21.788249 13235 raft_consensus.cc:1261] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f [term 1 FOLLOWER]: Refusing update from remote peer 08b377e3cc2b4a81a13d0d3f48c569d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:21.788793 13445 consensus_queue.cc:1035] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b0a86114f234e7bad2db7d66caac7ce" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:38:21.789171 13450 consensus_queue.cc:1035] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4889805ad4644e51a4b6e01b08a6477f" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 45887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:38:21.799502 13453 mvcc.cc:205] Tried to move back new op lower bound from 7334352903316774912 to 7334352903092334592. Current Snapshot: MvccSnapshot[applied={T|T < 7334352903316774912}]
I20260928 16:38:21.802132 13455 mvcc.cc:205] Tried to move back new op lower bound from 7334352903316774912 to 7334352903092334592. Current Snapshot: MvccSnapshot[applied={T|T < 7334352903316774912}]
I20260928 16:38:21.802246 13454 mvcc.cc:205] Tried to move back new op lower bound from 7334352903316774912 to 7334352903092334592. Current Snapshot: MvccSnapshot[applied={T|T < 7334352903316774912}]
I20260928 16:38:21.810941 12949 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:38:21.811124 12949 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:38:21.813912 12949 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4460c836dbb4e749dc15d4c6da1648c: Sending DeleteTablet for 3 replicas of tablet a85abeea8b294e7594e7112eb45a9557
I20260928 16:38:21.814749 13354 tablet_service.cc:1514] Processing DeleteTablet for tablet a85abeea8b294e7594e7112eb45a9557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:21 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:56034
I20260928 16:38:21.814958 13215 tablet_service.cc:1514] Processing DeleteTablet for tablet a85abeea8b294e7594e7112eb45a9557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:21 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:32852
I20260928 16:38:21.815229 13078 tablet_service.cc:1514] Processing DeleteTablet for tablet a85abeea8b294e7594e7112eb45a9557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:21 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:55162
I20260928 16:38:21.815670 13465 tablet_replica.cc:330] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6: stopping tablet replica
I20260928 16:38:21.816041 13465 raft_consensus.cc:2229] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:38:21.816248 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 13014
I20260928 16:38:21.816547 13465 raft_consensus.cc:2258] T a85abeea8b294e7594e7112eb45a9557 P 08b377e3cc2b4a81a13d0d3f48c569d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:21.818447 13463 tablet_replica.cc:330] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce: stopping tablet replica
I20260928 16:38:21.818779 13463 raft_consensus.cc:2229] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:38:21.819077 13463 raft_consensus.cc:2258] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:21.819440 13464 tablet_replica.cc:330] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f: stopping tablet replica
I20260928 16:38:21.819732 13464 raft_consensus.cc:2229] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:38:21.819993 13464 raft_consensus.cc:2258] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:21.820977 13463 ts_tablet_manager.cc:1905] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:38:21.821389 13464 ts_tablet_manager.cc:1905] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:38:21.824717 13464 ts_tablet_manager.cc:1918] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:38:21.824822 13464 log.cc:1196] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/a85abeea8b294e7594e7112eb45a9557
I20260928 16:38:21.825033 13463 ts_tablet_manager.cc:1918] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:38:21.825169 13464 ts_tablet_manager.cc:1939] T a85abeea8b294e7594e7112eb45a9557 P 4889805ad4644e51a4b6e01b08a6477f: Deleting consensus metadata
I20260928 16:38:21.825260 13463 log.cc:1196] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/a85abeea8b294e7594e7112eb45a9557
W20260928 16:38:21.825910 12933 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:39671 (error 108)
I20260928 16:38:21.826308 13463 ts_tablet_manager.cc:1939] T a85abeea8b294e7594e7112eb45a9557 P 6b0a86114f234e7bad2db7d66caac7ce: Deleting consensus metadata
I20260928 16:38:21.826390 12933 catalog_manager.cc:4841] TS 4889805ad4644e51a4b6e01b08a6477f (127.27.122.194:45887): tablet a85abeea8b294e7594e7112eb45a9557 (table test-table [id=de26bcd29b57453fb8666834572586da]) successfully deleted
I20260928 16:38:21.827026 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 13151
I20260928 16:38:21.827427 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:21.827077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:39671 (local address 127.0.0.1:55178)
0928 16:38:21.827231 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:21.827322 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 16:38:21.828608 12935 catalog_manager.cc:4841] TS 6b0a86114f234e7bad2db7d66caac7ce (127.27.122.195:38821): tablet a85abeea8b294e7594e7112eb45a9557 (table test-table [id=de26bcd29b57453fb8666834572586da]) successfully deleted
W20260928 16:38:21.829627 12933 catalog_manager.cc:4584] TS 08b377e3cc2b4a81a13d0d3f48c569d6 (127.27.122.193:39671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a85abeea8b294e7594e7112eb45a9557: Network error: Client connection negotiation failed: client connection to 127.27.122.193:39671: connect: Connection refused (error 111)
I20260928 16:38:21.834199 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 13288
I20260928 16:38:21.839944 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 12916
2026-09-28T16:38:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2387 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T16:38:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:21Z Disabled control of system clock
I20260928 16:38:21.869493 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:35623
--webserver_interface=127.27.122.254
--webserver_port=0
--builtin_ntp_servers=127.27.122.212:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:35623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:38:21.976234 13471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:21.976467 13471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:21.976501 13471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:21.980577 13471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:21.980630 13471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:21.980645 13471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:21.980662 13471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:21.980677 13471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:21.985419 13471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:35623
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:35623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:21.986379 13471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:21.987679 13471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:21.996028 13476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:21.996034 13479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:21.996065 13477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:21.996270 13471 server_base.cc:1034] running on GCE node
I20260928 16:38:21.996959 13471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:21.997956 13471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:21.999120 13471 hybrid_clock.cc:648] HybridClock initialized: now 1790613501999106 us; error 25 us; skew 500 ppm
I20260928 16:38:22.000887 13471 webserver.cc:460] Webserver started at http://127.27.122.254:33367/ using document root <none> and password file <none>
I20260928 16:38:22.001484 13471 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:22.001574 13471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:22.001799 13471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:22.003734 13471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "fd75cef045754476985a0661e6bdeb6a"
format_stamp: "Formatted at 2026-09-28 16:38:21 on dist-test-slave-g9sh"
server_key: "27ba9b1f2bf3a724121451453b7d8857"
server_key_iv: "b8a28cd3958c1d5ea10ab51db5e0fd49"
server_key_version: "encryptionkey@0"
I20260928 16:38:22.004271 13471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "fd75cef045754476985a0661e6bdeb6a"
format_stamp: "Formatted at 2026-09-28 16:38:21 on dist-test-slave-g9sh"
server_key: "27ba9b1f2bf3a724121451453b7d8857"
server_key_iv: "b8a28cd3958c1d5ea10ab51db5e0fd49"
server_key_version: "encryptionkey@0"
I20260928 16:38:22.007766 13471 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 16:38:22.010016 13485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:22.011155 13471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:38:22.011284 13471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "fd75cef045754476985a0661e6bdeb6a"
format_stamp: "Formatted at 2026-09-28 16:38:21 on dist-test-slave-g9sh"
server_key: "27ba9b1f2bf3a724121451453b7d8857"
server_key_iv: "b8a28cd3958c1d5ea10ab51db5e0fd49"
server_key_version: "encryptionkey@0"
I20260928 16:38:22.011389 13471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:22.025355 13471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:22.025951 13471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:22.026134 13471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:22.032644 13471 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:35623
I20260928 16:38:22.032660 13537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:35623 every 8 connection(s)
I20260928 16:38:22.033632 13471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:22.034372 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 13471
I20260928 16:38:22.034547 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:22.034865 28139 external_mini_cluster.cc:1428] Setting key 0d90b13501d98d0e383e7b6f1157a27d
I20260928 16:38:22.037364 13538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:22.043582 13538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a: Bootstrap starting.
I20260928 16:38:22.044044 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.036200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.27.122.254:35623)
0928 16:38:22.036787 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:22.036803 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:22.036837 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:38:22.037553 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:22.037571 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:22.037949 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:22.038222 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:22.038547 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.039497 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.040755 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.041285 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.041405 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.042274 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:22.042279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:22.042291 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:22.042695 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:22.042753 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:22.042773 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:22.042850 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:22.043020 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 16:38:22.043307 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":136,"threads_started":1}
I20260928 16:38:22.046021 13538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:22.046790 13538 log.cc:824] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:22.048583 13538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a: No bootstrap required, opened a new log
I20260928 16:38:22.052022 13538 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd75cef045754476985a0661e6bdeb6a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35623 } }
I20260928 16:38:22.052259 13538 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:22.052336 13538 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd75cef045754476985a0661e6bdeb6a, State: Initialized, Role: FOLLOWER
I20260928 16:38:22.052731 13538 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd75cef045754476985a0661e6bdeb6a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35623 } }
I20260928 16:38:22.052882 13538 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:22.052991 13538 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:22.053086 13538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:22.053915 13538 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd75cef045754476985a0661e6bdeb6a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35623 } }
I20260928 16:38:22.054239 13538 leader_election.cc:302] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd75cef045754476985a0661e6bdeb6a; no voters:
I20260928 16:38:22.054538 13538 leader_election.cc:288] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:22.054687 13543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:22.054919 13543 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [term 1 LEADER]: Becoming Leader. State: Replica: fd75cef045754476985a0661e6bdeb6a, State: Running, Role: LEADER
I20260928 16:38:22.055213 13543 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd75cef045754476985a0661e6bdeb6a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35623 } }
I20260928 16:38:22.055712 13538 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:22.056989 13544 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd75cef045754476985a0661e6bdeb6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd75cef045754476985a0661e6bdeb6a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35623 } } }
I20260928 16:38:22.057140 13544 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:22.057003 13545 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd75cef045754476985a0661e6bdeb6a. Latest consensus state: current_term: 1 leader_uuid: "fd75cef045754476985a0661e6bdeb6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd75cef045754476985a0661e6bdeb6a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35623 } } }
I20260928 16:38:22.057237 13545 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:22.058139 13552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:22.060662 13552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:22.066131 13552 catalog_manager.cc:1347] Generated new cluster ID: 3f185c2e932f4a1eb1e6a45f2eef4e39
I20260928 16:38:22.066193 13552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:22.076115 13552 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:22.077275 13552 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:22.084298 13552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a: Generated new TSK 0
I20260928 16:38:22.085004 13552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:22.089339 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35623
--builtin_ntp_servers=127.27.122.212:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:38:22.198829 13563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:22.199054 13563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:22.199091 13563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:22.202965 13563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:22.203007 13563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:22.203080 13563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:38:22.207588 13563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:22.208534 13563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:22.209873 13563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:38:22.219417 13563 server_base.cc:1034] running on GCE node
W20260928 16:38:22.219332 13569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:22.219373 13570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:22.219345 13572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:22.219918 13563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:22.220466 13563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:22.221617 13563 hybrid_clock.cc:648] HybridClock initialized: now 1790613502221602 us; error 26 us; skew 500 ppm
I20260928 16:38:22.223340 13563 webserver.cc:460] Webserver started at http://127.27.122.193:34801/ using document root <none> and password file <none>
I20260928 16:38:22.223909 13563 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:22.223971 13563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:22.224202 13563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:22.226239 13563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "3b00492b61a54c3ba19248078ce76329"
format_stamp: "Formatted at 2026-09-28 16:38:22 on dist-test-slave-g9sh"
server_key: "9e2a2f128104f54bf06c26a79dbccb06"
server_key_iv: "78876f3e38e7e8842665a1f813bf5335"
server_key_version: "encryptionkey@0"
I20260928 16:38:22.226810 13563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "3b00492b61a54c3ba19248078ce76329"
format_stamp: "Formatted at 2026-09-28 16:38:22 on dist-test-slave-g9sh"
server_key: "9e2a2f128104f54bf06c26a79dbccb06"
server_key_iv: "78876f3e38e7e8842665a1f813bf5335"
server_key_version: "encryptionkey@0"
I20260928 16:38:22.230257 13563 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:38:22.232522 13578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:22.233732 13563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:22.233861 13563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "3b00492b61a54c3ba19248078ce76329"
format_stamp: "Formatted at 2026-09-28 16:38:22 on dist-test-slave-g9sh"
server_key: "9e2a2f128104f54bf06c26a79dbccb06"
server_key_iv: "78876f3e38e7e8842665a1f813bf5335"
server_key_version: "encryptionkey@0"
I20260928 16:38:22.233963 13563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:22.246941 13563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:22.247526 13563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:22.247697 13563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:22.248203 13563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:22.249114 13563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:22.249179 13563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:22.249248 13563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:22.249287 13563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:22.257961 13563 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:40745
I20260928 16:38:22.258060 13691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:40745 every 8 connection(s)
I20260928 16:38:22.258818 13563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:38:22.265177 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 13563
I20260928 16:38:22.265307 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:38:22.265612 28139 external_mini_cluster.cc:1428] Setting key b4000538ab2edf61da460c8db796e12c
I20260928 16:38:22.267720 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35623
--builtin_ntp_servers=127.27.122.212:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:38:22.268050 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.260513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:56039 (local address 127.27.122.254:35623)
0928 16:38:22.260687 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:22.260691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:22.261716 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 16:38:22.262998 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:22.263001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:22.263055 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:22.263127 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:22.263897 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.264422 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.265904 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.266268 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.266354 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.267450 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:22.267456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:22.267461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:22.267478 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:22.267505 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:22.267519 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:22.267615 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:22.267752 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 16:38:22.267850 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:38:22.268805 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.260946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35623 (local address 127.27.122.193:56039)
0928 16:38:22.261573 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:22.261611 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:22.262825 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:22.263250 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:22.263256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:22.263333 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:22.263782 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.263794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.264525 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.264527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.265762 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.265774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.266421 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.266425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.266578 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.267199 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:38:22.267200 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:22.267239 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:38:22.267299 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:22.267620 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:22.267627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:22.267631 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:38:22.267722 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 16:38:22.268019 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":121,"threads_started":1}
I20260928 16:38:22.269997 13692 heartbeater.cc:347] Connected to a master server at 127.27.122.254:35623
I20260928 16:38:22.270228 13692 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:22.270710 13692 heartbeater.cc:511] Master 127.27.122.254:35623 requested a full tablet report, sending...
I20260928 16:38:22.272113 13502 ts_manager.cc:194] Registered new tserver with Master: 3b00492b61a54c3ba19248078ce76329 (127.27.122.193:40745)
I20260928 16:38:22.273283 13502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.122.193:56039
W20260928 16:38:22.375550 13695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:22.375793 13695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:22.375830 13695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:22.379834 13695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:22.379882 13695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:22.379957 13695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:38:22.384558 13695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:22.385608 13695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:22.386922 13695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:22.396044 13701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:22.396059 13703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:22.396250 13695 server_base.cc:1034] running on GCE node
W20260928 16:38:22.396137 13700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:22.396770 13695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:22.397360 13695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:22.398540 13695 hybrid_clock.cc:648] HybridClock initialized: now 1790613502398502 us; error 44 us; skew 500 ppm
I20260928 16:38:22.400321 13695 webserver.cc:460] Webserver started at http://127.27.122.194:45375/ using document root <none> and password file <none>
I20260928 16:38:22.400869 13695 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:22.400997 13695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:22.401253 13695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:22.403292 13695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "8228f68e2e3f42e586a561d6df90dc72"
format_stamp: "Formatted at 2026-09-28 16:38:22 on dist-test-slave-g9sh"
server_key: "df375c87a6224e41cb9b0353236035aa"
server_key_iv: "b8607cfeed2ca10325c73db128b76550"
server_key_version: "encryptionkey@0"
I20260928 16:38:22.403859 13695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "8228f68e2e3f42e586a561d6df90dc72"
format_stamp: "Formatted at 2026-09-28 16:38:22 on dist-test-slave-g9sh"
server_key: "df375c87a6224e41cb9b0353236035aa"
server_key_iv: "b8607cfeed2ca10325c73db128b76550"
server_key_version: "encryptionkey@0"
I20260928 16:38:22.407416 13695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 16:38:22.409833 13709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:22.410952 13695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:22.411087 13695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "8228f68e2e3f42e586a561d6df90dc72"
format_stamp: "Formatted at 2026-09-28 16:38:22 on dist-test-slave-g9sh"
server_key: "df375c87a6224e41cb9b0353236035aa"
server_key_iv: "b8607cfeed2ca10325c73db128b76550"
server_key_version: "encryptionkey@0"
I20260928 16:38:22.411188 13695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:22.442200 13695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:22.442889 13695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:22.443084 13695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:22.443701 13695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:22.444620 13695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:22.444689 13695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:22.444756 13695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:22.444801 13695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:22.453310 13695 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:34201
I20260928 16:38:22.453339 13826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:34201 every 8 connection(s)
I20260928 16:38:22.454249 13695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:38:22.464133 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 13695
I20260928 16:38:22.464277 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:38:22.464401 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.456018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:40557 (local address 127.27.122.254:35623)
0928 16:38:22.456158 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:22.456164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:22.457379 (+ 1215us) server_negotiation.cc:407] Connection header received
0928 16:38:22.459216 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:22.459220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:22.459282 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:22.459379 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:22.460552 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.461293 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.462526 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.462770 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.462881 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.463897 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:22.463903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:22.463906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:22.463918 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:22.463940 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:22.463949 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:22.464034 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:22.464191 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 16:38:22.464260 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:38:22.464617 28139 external_mini_cluster.cc:1428] Setting key f51d76ad8c08646be1b12979094a1f80
I20260928 16:38:22.465415 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.456508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35623 (local address 127.27.122.194:40557)
0928 16:38:22.457248 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:22.457287 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:22.459000 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:22.459593 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:22.459601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:22.459724 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:22.460409 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.460425 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.461436 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.461440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.462397 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.462407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.462921 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.462925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.463103 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.463636 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:38:22.463638 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:22.463676 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:38:22.463736 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:22.464041 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:22.464052 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:22.464058 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:38:22.464190 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 16:38:22.464451 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260928 16:38:22.466722 13827 heartbeater.cc:347] Connected to a master server at 127.27.122.254:35623
I20260928 16:38:22.466792 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35623
--builtin_ntp_servers=127.27.122.212:44967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:38:22.466958 13827 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:22.467536 13827 heartbeater.cc:511] Master 127.27.122.254:35623 requested a full tablet report, sending...
I20260928 16:38:22.468542 13502 ts_manager.cc:194] Registered new tserver with Master: 8228f68e2e3f42e586a561d6df90dc72 (127.27.122.194:34201)
I20260928 16:38:22.469143 13502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.122.194:40557
W20260928 16:38:22.574987 13830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:22.575220 13830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:22.575256 13830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:22.579130 13830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:22.579185 13830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:22.579237 13830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:38:22.583760 13830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:44967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:35623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:22.584749 13830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:22.586050 13830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:22.595338 13835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:22.595328 13836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:22.595361 13830 server_base.cc:1034] running on GCE node
W20260928 16:38:22.595329 13838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:22.595944 13830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:22.596570 13830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:22.597746 13830 hybrid_clock.cc:648] HybridClock initialized: now 1790613502597733 us; error 24 us; skew 500 ppm
I20260928 16:38:22.599484 13830 webserver.cc:460] Webserver started at http://127.27.122.195:41093/ using document root <none> and password file <none>
I20260928 16:38:22.600054 13830 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:22.600150 13830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:22.600365 13830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:22.602382 13830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "f5b0d79b70d64628b217fe0c31cb37e8"
format_stamp: "Formatted at 2026-09-28 16:38:22 on dist-test-slave-g9sh"
server_key: "d7464c0b0fef2c5705cc90b1cae31aef"
server_key_iv: "7bd10e75ccde63ffc18e5fa7b07cfa5d"
server_key_version: "encryptionkey@0"
I20260928 16:38:22.602957 13830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "f5b0d79b70d64628b217fe0c31cb37e8"
format_stamp: "Formatted at 2026-09-28 16:38:22 on dist-test-slave-g9sh"
server_key: "d7464c0b0fef2c5705cc90b1cae31aef"
server_key_iv: "7bd10e75ccde63ffc18e5fa7b07cfa5d"
server_key_version: "encryptionkey@0"
I20260928 16:38:22.606375 13830 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 16:38:22.608549 13844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:22.609738 13830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:38:22.609861 13830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "f5b0d79b70d64628b217fe0c31cb37e8"
format_stamp: "Formatted at 2026-09-28 16:38:22 on dist-test-slave-g9sh"
server_key: "d7464c0b0fef2c5705cc90b1cae31aef"
server_key_iv: "7bd10e75ccde63ffc18e5fa7b07cfa5d"
server_key_version: "encryptionkey@0"
I20260928 16:38:22.609961 13830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:22.638883 13830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:22.639570 13830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:22.639760 13830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:22.640344 13830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:22.641324 13830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:22.641389 13830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:22.641458 13830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:22.641505 13830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:22.650094 13830 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:37181
I20260928 16:38:22.650117 13961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:37181 every 8 connection(s)
I20260928 16:38:22.650950 13830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:38:22.653131 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 13830
I20260928 16:38:22.653249 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:38:22.653627 28139 external_mini_cluster.cc:1428] Setting key fd6c662125c5067d2fe6ba9be0c930c5
I20260928 16:38:22.660645 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.652646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:38959 (local address 127.27.122.254:35623)
0928 16:38:22.652798 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:22.652801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:22.654086 (+ 1285us) server_negotiation.cc:407] Connection header received
0928 16:38:22.655875 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:22.655878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:22.655935 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:22.656011 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:22.656964 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.657570 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.658769 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.659000 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.659108 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.660129 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:22.660133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:22.660136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:22.660148 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:22.660175 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:22.660183 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:22.660258 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:22.660411 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 16:38:22.660501 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:38:22.661468 13964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.653107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:35623 (local address 127.27.122.195:38959)
0928 16:38:22.653919 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:22.653982 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:22.655622 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:22.656182 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:22.656189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:22.656271 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:22.656793 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.656831 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.657703 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.657706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.658636 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.658646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.659147 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.659151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.659325 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.659869 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:38:22.659871 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:22.659910 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:38:22.659982 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:22.660273 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:22.660283 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:22.660292 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 16:38:22.660411 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 16:38:22.660677 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":149,"threads_started":1}
I20260928 16:38:22.662441 13962 heartbeater.cc:347] Connected to a master server at 127.27.122.254:35623
I20260928 16:38:22.662644 13962 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:22.663084 13962 heartbeater.cc:511] Master 127.27.122.254:35623 requested a full tablet report, sending...
I20260928 16:38:22.663977 13502 ts_manager.cc:194] Registered new tserver with Master: f5b0d79b70d64628b217fe0c31cb37e8 (127.27.122.195:37181)
I20260928 16:38:22.664508 13502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.122.195:38959
I20260928 16:38:22.667500 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:38:22.672614 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.669421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.27.122.254:35623)
0928 16:38:22.669548 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:22.669551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:22.669705 (+ 154us) server_negotiation.cc:407] Connection header received
0928 16:38:22.669805 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:22.669807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:22.669844 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:22.669903 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:22.670097 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.670577 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.671337 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.671552 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.671626 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.672284 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:22.672287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:22.672289 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:22.672300 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:22.672316 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:22.672323 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:22.672379 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:22.672464 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 16:38:22.672523 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 16:38:22.675127 13502 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:38:22.678153 13502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:38:22.691772 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.686125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.27.122.194:34201)
0928 16:38:22.686460 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:22.686470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:22.686493 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:38:22.686624 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:22.686628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:22.686826 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:22.687368 (+ 542us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:22.687511 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.688631 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.690226 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.691086 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.691208 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.691380 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:22.691514 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 16:38:22.691608 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":145,"threads_started":1}
I20260928 16:38:22.691787 13975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.686685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:37181 (local address 127.0.0.1:59620)
0928 16:38:22.686962 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:22.686977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:22.687054 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:22.687854 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:22.687857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:22.687875 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:22.687963 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.687968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.689304 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.689307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.690225 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.690235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.691318 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.691322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.691398 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.691414 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 16:38:22.691458 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 16:38:22.691618 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":3,"thread_start_us":113,"threads_started":1}
I20260928 16:38:22.691788 13973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.685963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:34201 (local address 127.0.0.1:46314)
0928 16:38:22.686298 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:22.686340 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:22.686455 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:22.687027 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:22.687033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:22.687060 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:22.687189 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.687198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.688745 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.688748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.690105 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.690114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.691202 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.691206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.691337 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.691379 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:38:22.691482 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 16:38:22.691628 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":101,"thread_start_us":107,"threads_started":1}
I20260928 16:38:22.692435 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.687043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40745 (local address 127.0.0.1:33592)
0928 16:38:22.687385 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:22.687401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:22.687480 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:22.688252 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:22.688255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:22.688275 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:22.688369 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.688374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.689886 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.689890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.690966 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.690976 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.692067 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.692071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.692166 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.692187 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:38:22.692250 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 16:38:22.692319 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260928 16:38:22.692538 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.687466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33592 (local address 127.27.122.193:40745)
0928 16:38:22.687807 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:22.687819 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:22.687840 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:38:22.687912 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:22.687916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:22.688101 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:22.688491 (+ 390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:22.688611 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.689766 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.691115 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.691902 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.692074 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.692247 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:22.692294 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 16:38:22.692381 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":146,"threads_started":1}
I20260928 16:38:22.693408 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.686843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.27.122.195:37181)
0928 16:38:22.687398 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:22.687411 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:22.687434 (+ 23us) server_negotiation.cc:407] Connection header received
0928 16:38:22.687506 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:22.687511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:22.687698 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:22.688110 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:22.688233 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.689194 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.690383 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.691196 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.693022 (+ 1826us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.693135 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:22.693172 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 16:38:22.693258 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":104,"threads_started":1}
I20260928 16:38:22.695025 13759 tablet_service.cc:1467] Processing CreateTablet for tablet a36658c036c3439fbffdca2e04092b8a (DEFAULT_TABLE table=test-table [id=aff27ff7252e499db96d19cc7ef47400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:22.695271 13626 tablet_service.cc:1467] Processing CreateTablet for tablet a36658c036c3439fbffdca2e04092b8a (DEFAULT_TABLE table=test-table [id=aff27ff7252e499db96d19cc7ef47400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:22.696015 13759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36658c036c3439fbffdca2e04092b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:22.695791 13893 tablet_service.cc:1467] Processing CreateTablet for tablet a36658c036c3439fbffdca2e04092b8a (DEFAULT_TABLE table=test-table [id=aff27ff7252e499db96d19cc7ef47400]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:22.696300 13626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36658c036c3439fbffdca2e04092b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:22.696723 13893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a36658c036c3439fbffdca2e04092b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:22.702768 13979 tablet_bootstrap.cc:492] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72: Bootstrap starting.
I20260928 16:38:22.703850 13981 tablet_bootstrap.cc:492] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8: Bootstrap starting.
I20260928 16:38:22.704870 13979 tablet_bootstrap.cc:654] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:22.705611 13981 tablet_bootstrap.cc:654] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:22.705631 13979 log.cc:824] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:22.706166 13980 tablet_bootstrap.cc:492] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329: Bootstrap starting.
I20260928 16:38:22.706388 13981 log.cc:824] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:22.708019 13981 tablet_bootstrap.cc:492] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8: No bootstrap required, opened a new log
I20260928 16:38:22.708200 13981 ts_tablet_manager.cc:1397] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 16:38:22.708828 13979 tablet_bootstrap.cc:492] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72: No bootstrap required, opened a new log
I20260928 16:38:22.709043 13979 ts_tablet_manager.cc:1397] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:38:22.711261 13981 raft_consensus.cc:348] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8228f68e2e3f42e586a561d6df90dc72" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 34201 } } peers { permanent_uuid: "f5b0d79b70d64628b217fe0c31cb37e8" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37181 } } peers { permanent_uuid: "3b00492b61a54c3ba19248078ce76329" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40745 } }
I20260928 16:38:22.711464 13981 raft_consensus.cc:374] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:22.711520 13981 raft_consensus.cc:729] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b0d79b70d64628b217fe0c31cb37e8, State: Initialized, Role: FOLLOWER
I20260928 16:38:22.711956 13981 consensus_queue.cc:260] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8228f68e2e3f42e586a561d6df90dc72" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 34201 } } peers { permanent_uuid: "f5b0d79b70d64628b217fe0c31cb37e8" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37181 } } peers { permanent_uuid: "3b00492b61a54c3ba19248078ce76329" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40745 } }
I20260928 16:38:22.712152 13980 tablet_bootstrap.cc:654] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:22.712163 13979 raft_consensus.cc:348] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8228f68e2e3f42e586a561d6df90dc72" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 34201 } } peers { permanent_uuid: "f5b0d79b70d64628b217fe0c31cb37e8" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37181 } } peers { permanent_uuid: "3b00492b61a54c3ba19248078ce76329" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40745 } }
I20260928 16:38:22.712414 13979 raft_consensus.cc:374] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:22.712476 13979 raft_consensus.cc:729] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8228f68e2e3f42e586a561d6df90dc72, State: Initialized, Role: FOLLOWER
I20260928 16:38:22.712873 13979 consensus_queue.cc:260] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8228f68e2e3f42e586a561d6df90dc72" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 34201 } } peers { permanent_uuid: "f5b0d79b70d64628b217fe0c31cb37e8" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37181 } } peers { permanent_uuid: "3b00492b61a54c3ba19248078ce76329" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40745 } }
I20260928 16:38:22.712972 13981 ts_tablet_manager.cc:1428] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 16:38:22.712996 13980 log.cc:824] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:22.713325 13962 heartbeater.cc:503] Master 127.27.122.254:35623 was elected leader, sending a full tablet report...
I20260928 16:38:22.713688 13827 heartbeater.cc:503] Master 127.27.122.254:35623 was elected leader, sending a full tablet report...
I20260928 16:38:22.713801 13979 ts_tablet_manager.cc:1428] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:38:22.715310 13980 tablet_bootstrap.cc:492] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329: No bootstrap required, opened a new log
I20260928 16:38:22.715484 13980 ts_tablet_manager.cc:1397] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260928 16:38:22.719233 13980 raft_consensus.cc:348] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8228f68e2e3f42e586a561d6df90dc72" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 34201 } } peers { permanent_uuid: "f5b0d79b70d64628b217fe0c31cb37e8" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37181 } } peers { permanent_uuid: "3b00492b61a54c3ba19248078ce76329" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40745 } }
I20260928 16:38:22.719412 13980 raft_consensus.cc:374] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:22.719480 13980 raft_consensus.cc:729] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b00492b61a54c3ba19248078ce76329, State: Initialized, Role: FOLLOWER
I20260928 16:38:22.719843 13980 consensus_queue.cc:260] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8228f68e2e3f42e586a561d6df90dc72" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 34201 } } peers { permanent_uuid: "f5b0d79b70d64628b217fe0c31cb37e8" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37181 } } peers { permanent_uuid: "3b00492b61a54c3ba19248078ce76329" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40745 } }
I20260928 16:38:22.720413 13692 heartbeater.cc:503] Master 127.27.122.254:35623 was elected leader, sending a full tablet report...
I20260928 16:38:22.720611 13980 ts_tablet_manager.cc:1428] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260928 16:38:22.759915 13693 tablet.cc:2369] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:38:22.901867 13963 tablet.cc:2369] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:38:22.939556 13987 raft_consensus.cc:482] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:38:22.939795 13987 raft_consensus.cc:504] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8228f68e2e3f42e586a561d6df90dc72" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 34201 } } peers { permanent_uuid: "f5b0d79b70d64628b217fe0c31cb37e8" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37181 } } peers { permanent_uuid: "3b00492b61a54c3ba19248078ce76329" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40745 } }
I20260928 16:38:22.940975 13987 leader_election.cc:288] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8228f68e2e3f42e586a561d6df90dc72 (127.27.122.194:34201), f5b0d79b70d64628b217fe0c31cb37e8 (127.27.122.195:37181)
I20260928 16:38:22.945016 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.941786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:58751 (local address 127.27.122.195:37181)
0928 16:38:22.941912 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:22.941917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:22.941960 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:38:22.942074 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:22.942077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:22.942135 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:22.942212 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:22.942505 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.942977 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.944056 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.944440 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.944558 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.944620 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:22.944792 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 16:38:22.944873 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:38:22.945015 13990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.941608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:37181 (local address 127.27.122.193:58751)
0928 16:38:22.941888 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:22.941900 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:22.941993 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:22.942229 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:22.942231 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:22.942244 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:22.942394 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.942399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.943109 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.943111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.943932 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.943940 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.944568 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.944572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.944674 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.944698 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:38:22.944759 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 16:38:22.944813 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":5,"thread_start_us":89,"threads_started":1}
I20260928 16:38:22.945106 13974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.941331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:45755 (local address 127.27.122.194:34201)
0928 16:38:22.941479 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:22.941484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:22.941665 (+ 181us) server_negotiation.cc:407] Connection header received
0928 16:38:22.941841 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:22.941846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:22.941900 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:22.941952 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:22.942404 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.942953 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.944071 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.944456 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.944585 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.944636 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:22.944879 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 16:38:22.944971 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:38:22.945477 13989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.941247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:34201 (local address 127.27.122.193:45755)
0928 16:38:22.941514 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:22.941528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:22.941696 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:22.942110 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:22.942112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:22.942131 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:22.942278 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.942287 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.943070 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.943073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.943931 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:22.943940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.944631 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.944637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:22.944728 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.944770 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 16:38:22.944847 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:38:22.944931 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":23,"thread_start_us":95,"threads_started":1}
I20260928 16:38:22.945788 13914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36658c036c3439fbffdca2e04092b8a" candidate_uuid: "3b00492b61a54c3ba19248078ce76329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b0d79b70d64628b217fe0c31cb37e8" is_pre_election: true
I20260928 16:38:22.946139 13914 raft_consensus.cc:2454] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b00492b61a54c3ba19248078ce76329 in term 0.
I20260928 16:38:22.946235 13780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36658c036c3439fbffdca2e04092b8a" candidate_uuid: "3b00492b61a54c3ba19248078ce76329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8228f68e2e3f42e586a561d6df90dc72" is_pre_election: true
I20260928 16:38:22.946579 13780 raft_consensus.cc:2454] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b00492b61a54c3ba19248078ce76329 in term 0.
I20260928 16:38:22.946647 13582 leader_election.cc:302] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b00492b61a54c3ba19248078ce76329, f5b0d79b70d64628b217fe0c31cb37e8; no voters:
I20260928 16:38:22.946950 13987 raft_consensus.cc:2781] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:38:22.947052 13987 raft_consensus.cc:482] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:38:22.947119 13987 raft_consensus.cc:3037] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:22.948135 13987 raft_consensus.cc:504] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8228f68e2e3f42e586a561d6df90dc72" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 34201 } } peers { permanent_uuid: "f5b0d79b70d64628b217fe0c31cb37e8" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37181 } } peers { permanent_uuid: "3b00492b61a54c3ba19248078ce76329" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40745 } }
I20260928 16:38:22.948460 13987 leader_election.cc:288] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [CANDIDATE]: Term 1 election: Requested vote from peers 8228f68e2e3f42e586a561d6df90dc72 (127.27.122.194:34201), f5b0d79b70d64628b217fe0c31cb37e8 (127.27.122.195:37181)
I20260928 16:38:22.948865 13780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36658c036c3439fbffdca2e04092b8a" candidate_uuid: "3b00492b61a54c3ba19248078ce76329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8228f68e2e3f42e586a561d6df90dc72"
I20260928 16:38:22.948942 13914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a36658c036c3439fbffdca2e04092b8a" candidate_uuid: "3b00492b61a54c3ba19248078ce76329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5b0d79b70d64628b217fe0c31cb37e8"
I20260928 16:38:22.949012 13780 raft_consensus.cc:3037] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:22.949093 13914 raft_consensus.cc:3037] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:22.950040 13780 raft_consensus.cc:2454] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b00492b61a54c3ba19248078ce76329 in term 1.
I20260928 16:38:22.950181 13914 raft_consensus.cc:2454] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b00492b61a54c3ba19248078ce76329 in term 1.
I20260928 16:38:22.950351 13582 leader_election.cc:302] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b00492b61a54c3ba19248078ce76329, 8228f68e2e3f42e586a561d6df90dc72; no voters:
I20260928 16:38:22.950538 13987 raft_consensus.cc:2781] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:22.950775 13987 raft_consensus.cc:686] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [term 1 LEADER]: Becoming Leader. State: Replica: 3b00492b61a54c3ba19248078ce76329, State: Running, Role: LEADER
I20260928 16:38:22.951089 13987 consensus_queue.cc:237] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8228f68e2e3f42e586a561d6df90dc72" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 34201 } } peers { permanent_uuid: "f5b0d79b70d64628b217fe0c31cb37e8" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37181 } } peers { permanent_uuid: "3b00492b61a54c3ba19248078ce76329" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40745 } }
I20260928 16:38:22.954308 13502 catalog_manager.cc:5495] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b00492b61a54c3ba19248078ce76329 (127.27.122.193). New cstate: current_term: 1 leader_uuid: "3b00492b61a54c3ba19248078ce76329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8228f68e2e3f42e586a561d6df90dc72" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 34201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5b0d79b70d64628b217fe0c31cb37e8" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b00492b61a54c3ba19248078ce76329" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 40745 } health_report { overall_health: HEALTHY } } }
W20260928 16:38:22.955241 13828 tablet.cc:2369] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:38:22.982331 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:22.978824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33594 (local address 127.27.122.193:40745)
0928 16:38:22.978976 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:22.978983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:22.979000 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:38:22.979092 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:22.979094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:22.979148 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:22.979269 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:38:22.979510 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.980011 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.980879 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:22.981154 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:22.981265 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:22.981567 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:38:22.981958 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:38:22.982100 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:22.982138 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 16:38:22.982190 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:38:22.988086 13914 raft_consensus.cc:1261] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8 [term 1 FOLLOWER]: Refusing update from remote peer 3b00492b61a54c3ba19248078ce76329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:22.988075 13780 raft_consensus.cc:1261] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72 [term 1 FOLLOWER]: Refusing update from remote peer 3b00492b61a54c3ba19248078ce76329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:22.988826 13991 consensus_queue.cc:1035] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5b0d79b70d64628b217fe0c31cb37e8" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 37181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:38:22.989065 13987 consensus_queue.cc:1035] T a36658c036c3439fbffdca2e04092b8a P 3b00492b61a54c3ba19248078ce76329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8228f68e2e3f42e586a561d6df90dc72" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 34201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:38:22.996948 13994 mvcc.cc:205] Tried to move back new op lower bound from 7334352908232040448 to 7334352908090073088. Current Snapshot: MvccSnapshot[applied={T|T < 7334352908232040448 or (T in {7334352908232040448})}]
I20260928 16:38:23.004088 13501 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:38:23.004252 13501 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:38:23.006414 13501 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd75cef045754476985a0661e6bdeb6a: Sending DeleteTablet for 3 replicas of tablet a36658c036c3439fbffdca2e04092b8a
I20260928 16:38:23.007236 13893 tablet_service.cc:1514] Processing DeleteTablet for tablet a36658c036c3439fbffdca2e04092b8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:22 UTC) from {username='slave'} at 127.0.0.1:59620
I20260928 16:38:23.007232 13759 tablet_service.cc:1514] Processing DeleteTablet for tablet a36658c036c3439fbffdca2e04092b8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:22 UTC) from {username='slave'} at 127.0.0.1:46314
I20260928 16:38:23.007718 14003 tablet_replica.cc:330] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72: stopping tablet replica
I20260928 16:38:23.007702 13626 tablet_service.cc:1514] Processing DeleteTablet for tablet a36658c036c3439fbffdca2e04092b8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:22 UTC) from {username='slave'} at 127.0.0.1:33592
I20260928 16:38:23.007957 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 13563
I20260928 16:38:23.008016 14003 raft_consensus.cc:2229] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:38:23.008312 14003 raft_consensus.cc:2258] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:23.008790 14004 tablet_replica.cc:330] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8: stopping tablet replica
I20260928 16:38:23.009065 14004 raft_consensus.cc:2229] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:38:23.009334 14004 raft_consensus.cc:2258] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:23.010613 14003 ts_tablet_manager.cc:1905] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:38:23.011291 14004 ts_tablet_manager.cc:1905] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:38:23.013473 14003 ts_tablet_manager.cc:1918] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:38:23.013576 14003 log.cc:1196] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/a36658c036c3439fbffdca2e04092b8a
I20260928 16:38:23.013901 14003 ts_tablet_manager.cc:1939] T a36658c036c3439fbffdca2e04092b8a P 8228f68e2e3f42e586a561d6df90dc72: Deleting consensus metadata
I20260928 16:38:23.014134 14004 ts_tablet_manager.cc:1918] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:38:23.014223 14004 log.cc:1196] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/a36658c036c3439fbffdca2e04092b8a
I20260928 16:38:23.014503 14004 ts_tablet_manager.cc:1939] T a36658c036c3439fbffdca2e04092b8a P f5b0d79b70d64628b217fe0c31cb37e8: Deleting consensus metadata
I20260928 16:38:23.015187 13489 catalog_manager.cc:4841] TS 8228f68e2e3f42e586a561d6df90dc72 (127.27.122.194:34201): tablet a36658c036c3439fbffdca2e04092b8a (table test-table [id=aff27ff7252e499db96d19cc7ef47400]) successfully deleted
I20260928 16:38:23.015506 13489 catalog_manager.cc:4841] TS f5b0d79b70d64628b217fe0c31cb37e8 (127.27.122.195:37181): tablet a36658c036c3439fbffdca2e04092b8a (table test-table [id=aff27ff7252e499db96d19cc7ef47400]) successfully deleted
W20260928 16:38:23.016062 13489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:40745 (error 108)
I20260928 16:38:23.016314 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 13695
I20260928 16:38:23.017253 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:23.016996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:40745 (local address 127.0.0.1:33610)
0928 16:38:23.017123 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:23.017189 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:40745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 16:38:23.017378 13489 catalog_manager.cc:4584] TS 3b00492b61a54c3ba19248078ce76329 (127.27.122.193:40745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a36658c036c3439fbffdca2e04092b8a: Network error: Client connection negotiation failed: client connection to 127.27.122.193:40745: connect: Connection refused (error 111)
I20260928 16:38:23.022797 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 13830
I20260928 16:38:23.028298 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 13471
2026-09-28T16:38:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1187 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:23 dist-test-slave-g9sh krb5kdc[14009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:23 dist-test-slave-g9sh krb5kdc[14009](info): set up 2 sockets
Sep 28 16:38:23 dist-test-slave-g9sh krb5kdc[14009](info): commencing operation
krb5kdc: starting...
W20260928 16:38:24.063750 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:24 dist-test-slave-g9sh krb5kdc[14009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:38:24.181264 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:33803
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:33803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:24.288393 14022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:24.288640 14022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:24.288678 14022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:24.292672 14022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:24.292725 14022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:24.292744 14022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:24.292761 14022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:24.292776 14022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:24.297876 14022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:33803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:33803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:24.298827 14022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:24.299845 14022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:24.308046 14028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:24.308089 14022 server_base.cc:1034] running on GCE node
W20260928 16:38:24.308246 14030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:24.308398 14027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:24.308983 14022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:24.309935 14022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:24.311105 14022 hybrid_clock.cc:648] HybridClock initialized: now 1790613504311083 us; error 29 us; skew 500 ppm
Sep 28 16:38:24 dist-test-slave-g9sh krb5kdc[14009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613504, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:24.313917 14022 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:38:24.314816 14022 webserver.cc:460] Webserver started at http://127.27.122.254:43901/ using document root <none> and password file <none>
I20260928 16:38:24.315335 14022 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:24.315393 14022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:24.315626 14022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:24.317655 14022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "b11bc9556d764c34843543de249b6431"
format_stamp: "Formatted at 2026-09-28 16:38:24 on dist-test-slave-g9sh"
server_key: "4bb3430ecbf75a69f9ced72e8b75af00"
server_key_iv: "aba398219cc0f3d392d89708062865d5"
server_key_version: "encryptionkey@0"
I20260928 16:38:24.318187 14022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "b11bc9556d764c34843543de249b6431"
format_stamp: "Formatted at 2026-09-28 16:38:24 on dist-test-slave-g9sh"
server_key: "4bb3430ecbf75a69f9ced72e8b75af00"
server_key_iv: "aba398219cc0f3d392d89708062865d5"
server_key_version: "encryptionkey@0"
I20260928 16:38:24.321739 14022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:38:24.324015 14037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:24.325583 14022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:38:24.325716 14022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "b11bc9556d764c34843543de249b6431"
format_stamp: "Formatted at 2026-09-28 16:38:24 on dist-test-slave-g9sh"
server_key: "4bb3430ecbf75a69f9ced72e8b75af00"
server_key_iv: "aba398219cc0f3d392d89708062865d5"
server_key_version: "encryptionkey@0"
I20260928 16:38:24.325820 14022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:24.390954 14022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:24.394276 14022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:24.394495 14022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:24.402001 14022 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:33803
I20260928 16:38:24.402016 14091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:33803 every 8 connection(s)
I20260928 16:38:24.402920 14022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:24.405691 14092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:24.406962 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 14022
I20260928 16:38:24.407106 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:24.407394 28139 external_mini_cluster.cc:1428] Setting key 61996924e1dd7043d3e4fd04a15f852a
I20260928 16:38:24.412215 14092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431: Bootstrap starting.
I20260928 16:38:24.414775 14092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:24.415520 14092 log.cc:824] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:38:24 dist-test-slave-g9sh krb5kdc[14009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:24.417780 14092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431: No bootstrap required, opened a new log
I20260928 16:38:24.422456 14092 raft_consensus.cc:348] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11bc9556d764c34843543de249b6431" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 33803 } }
I20260928 16:38:24.422672 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:24.408698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51672 (local address 127.27.122.254:33803)
0928 16:38:24.409117 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:24.409135 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:24.409176 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:38:24.409976 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:24.410019 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:24.410430 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:24.410816 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:24.411434 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:24.412861 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:24.413811 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:24.414477 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:24.414640 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:24.417577 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:24.417583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:24.417598 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:24.417627 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:24.419574 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:24.419993 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:24.420002 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:24.420009 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:24.420177 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:24.420451 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:24.420457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:24.420460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:24.420978 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:24.421174 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:24.421416 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 16:38:24.421717 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":99,"threads_started":1}
I20260928 16:38:24.422755 14092 raft_consensus.cc:374] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:24.422827 14092 raft_consensus.cc:729] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b11bc9556d764c34843543de249b6431, State: Initialized, Role: FOLLOWER
I20260928 16:38:24.423259 14092 consensus_queue.cc:260] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11bc9556d764c34843543de249b6431" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 33803 } }
I20260928 16:38:24.423390 14092 raft_consensus.cc:388] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:24.423480 14092 raft_consensus.cc:482] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:24.423581 14092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:24.424409 14092 raft_consensus.cc:504] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11bc9556d764c34843543de249b6431" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 33803 } }
I20260928 16:38:24.424770 14092 leader_election.cc:302] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b11bc9556d764c34843543de249b6431; no voters:
I20260928 16:38:24.425132 14092 leader_election.cc:288] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:24.425243 14097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:24.425482 14097 raft_consensus.cc:686] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [term 1 LEADER]: Becoming Leader. State: Replica: b11bc9556d764c34843543de249b6431, State: Running, Role: LEADER
I20260928 16:38:24.425766 14097 consensus_queue.cc:237] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11bc9556d764c34843543de249b6431" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 33803 } }
I20260928 16:38:24.426316 14092 sys_catalog.cc:564] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:24.427421 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b11bc9556d764c34843543de249b6431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11bc9556d764c34843543de249b6431" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 33803 } } }
I20260928 16:38:24.427546 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:24.427847 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b11bc9556d764c34843543de249b6431. Latest consensus state: current_term: 1 leader_uuid: "b11bc9556d764c34843543de249b6431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11bc9556d764c34843543de249b6431" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 33803 } } }
I20260928 16:38:24.427958 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:24.427925 14106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:24.431028 14106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:24.436367 14106 catalog_manager.cc:1347] Generated new cluster ID: 8b2723235b264c049ad4da407560b0f8
I20260928 16:38:24.436427 14106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:24.457904 14106 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:24.459054 14106 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:24.463869 14106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431: Generated new TSK 0
I20260928 16:38:24.464583 14106 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:38:24.501761 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:33803
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:24.613199 14118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:24.613431 14118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:24.613471 14118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:24.617388 14118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:24.617434 14118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:24.617522 14118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:38:24.622530 14118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:33803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:24.623452 14118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:24.624416 14118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:24.634052 14123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:24.634094 14124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:24.634121 14126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:24.634313 14118 server_base.cc:1034] running on GCE node
I20260928 16:38:24.634789 14118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:24.635339 14118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:24.636497 14118 hybrid_clock.cc:648] HybridClock initialized: now 1790613504636472 us; error 44 us; skew 500 ppm
Sep 28 16:38:24 dist-test-slave-g9sh krb5kdc[14009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613504, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:24.639278 14118 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:38:24.640084 14118 webserver.cc:460] Webserver started at http://127.27.122.193:33803/ using document root <none> and password file <none>
I20260928 16:38:24.640588 14118 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:24.640646 14118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:24.640805 14118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:24.642762 14118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "4114afe5ea9f4310933143efc68a1836"
format_stamp: "Formatted at 2026-09-28 16:38:24 on dist-test-slave-g9sh"
server_key: "a52ccf68ad4c869dcb9a8cefa0623604"
server_key_iv: "f7ccf9c3993e46c2b6d8823891df0e3f"
server_key_version: "encryptionkey@0"
I20260928 16:38:24.643261 14118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "4114afe5ea9f4310933143efc68a1836"
format_stamp: "Formatted at 2026-09-28 16:38:24 on dist-test-slave-g9sh"
server_key: "a52ccf68ad4c869dcb9a8cefa0623604"
server_key_iv: "f7ccf9c3993e46c2b6d8823891df0e3f"
server_key_version: "encryptionkey@0"
I20260928 16:38:24.646894 14118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:38:24.649492 14134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:24.650846 14118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:24.650985 14118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "4114afe5ea9f4310933143efc68a1836"
format_stamp: "Formatted at 2026-09-28 16:38:24 on dist-test-slave-g9sh"
server_key: "a52ccf68ad4c869dcb9a8cefa0623604"
server_key_iv: "f7ccf9c3993e46c2b6d8823891df0e3f"
server_key_version: "encryptionkey@0"
I20260928 16:38:24.651093 14118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:24.662892 14118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:24.665616 14118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:24.665803 14118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:24.666344 14118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:24.667217 14118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:24.667337 14118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:24.667452 14118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:24.667495 14118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:24.676527 14118 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:41503
I20260928 16:38:24.676573 14249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:41503 every 8 connection(s)
I20260928 16:38:24.677444 14118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:38:24.677839 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 14118
I20260928 16:38:24.678046 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:38:24.678386 28139 external_mini_cluster.cc:1428] Setting key 8f06e5428766acb7e1b0a6c58a481c2e
Sep 28 16:38:24 dist-test-slave-g9sh krb5kdc[14009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613504, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
I20260928 16:38:24.693653 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:24.679367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:34967 (local address 127.27.122.254:33803)
0928 16:38:24.679529 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:24.679534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:24.680484 (+ 950us) server_negotiation.cc:407] Connection header received
0928 16:38:24.682353 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:24.682359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:24.682424 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:24.682536 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:24.683633 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:24.684237 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:24.686151 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:24.686409 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:24.686501 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:24.689464 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:24.689467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:24.689470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:24.689495 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:24.691110 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:24.691726 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:24.691732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:24.691734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:24.691820 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:24.692228 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:24.692232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:24.692234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:24.692527 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:24.692649 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:24.693218 (+ 569us) server_negotiation.cc:299] Negotiation successful
0928 16:38:24.693371 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:38:24.694254 14252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:24.679804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:33803 (local address 127.27.122.193:34967)
0928 16:38:24.680291 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:24.680348 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:24.682125 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:24.682681 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:24.682687 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:24.683034 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:24.683486 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:24.683499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:24.684377 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:24.684383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:24.686008 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:24.686022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:24.686585 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:24.686592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:24.686831 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:24.687467 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:24.687486 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:24.689277 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:24.691243 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:24.691249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:24.691296 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:24.691609 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:24.691939 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:24.691942 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:24.691944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:24.692111 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:24.692652 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:24.692658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:24.692879 (+ 221us) client_negotiation.cc:769] Sending connection context
0928 16:38:24.693121 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 16:38:24.693363 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":119,"threads_started":1}
I20260928 16:38:24.695632 14250 heartbeater.cc:347] Connected to a master server at 127.27.122.254:33803
I20260928 16:38:24.695881 14250 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:24.696367 14250 heartbeater.cc:511] Master 127.27.122.254:33803 requested a full tablet report, sending...
I20260928 16:38:24.697973 14055 ts_manager.cc:194] Registered new tserver with Master: 4114afe5ea9f4310933143efc68a1836 (127.27.122.193:41503)
I20260928 16:38:24.699177 14055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:34967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:38:24.707998 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:33803
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:24.816562 14255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:24.816782 14255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:24.816821 14255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:24.820729 14255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:24.820780 14255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:24.820838 14255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:38:24.826027 14255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:33803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:24.826927 14255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:24.827983 14255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:24.837452 14263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:24.837466 14260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:24.837410 14261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:24.837622 14255 server_base.cc:1034] running on GCE node
I20260928 16:38:24.837996 14255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:24.838598 14255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:24.839761 14255 hybrid_clock.cc:648] HybridClock initialized: now 1790613504839747 us; error 25 us; skew 500 ppm
Sep 28 16:38:24 dist-test-slave-g9sh krb5kdc[14009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613504, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:24.842585 14255 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:38:24.843346 14255 webserver.cc:460] Webserver started at http://127.27.122.194:35695/ using document root <none> and password file <none>
I20260928 16:38:24.843847 14255 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:24.843918 14255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:24.844081 14255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:24.846025 14255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "67f8e8863e78440da20ec45170de9a9a"
format_stamp: "Formatted at 2026-09-28 16:38:24 on dist-test-slave-g9sh"
server_key: "61186758c276dd2726d2e3071b226a41"
server_key_iv: "7eec0f85eccb040c931215205106db69"
server_key_version: "encryptionkey@0"
I20260928 16:38:24.846513 14255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "67f8e8863e78440da20ec45170de9a9a"
format_stamp: "Formatted at 2026-09-28 16:38:24 on dist-test-slave-g9sh"
server_key: "61186758c276dd2726d2e3071b226a41"
server_key_iv: "7eec0f85eccb040c931215205106db69"
server_key_version: "encryptionkey@0"
I20260928 16:38:24.850090 14255 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 16:38:24.852277 14270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:24.853474 14255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:38:24.853575 14255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "67f8e8863e78440da20ec45170de9a9a"
format_stamp: "Formatted at 2026-09-28 16:38:24 on dist-test-slave-g9sh"
server_key: "61186758c276dd2726d2e3071b226a41"
server_key_iv: "7eec0f85eccb040c931215205106db69"
server_key_version: "encryptionkey@0"
I20260928 16:38:24.853698 14255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:24.881807 14255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:24.884600 14255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:24.884799 14255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:24.885406 14255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:24.886234 14255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:24.886279 14255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:24.886349 14255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:24.886392 14255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:24.895049 14255 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:41409
I20260928 16:38:24.895092 14385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:41409 every 8 connection(s)
I20260928 16:38:24.895954 14255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:38:24.903918 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 14255
I20260928 16:38:24.904062 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:38:24.904352 28139 external_mini_cluster.cc:1428] Setting key 4b324d72e85cf70d0cf8c92d3108406b
Sep 28 16:38:24 dist-test-slave-g9sh krb5kdc[14009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613504, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:24.909693 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:24.897675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:45427 (local address 127.27.122.254:33803)
0928 16:38:24.897811 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:24.897814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:24.898633 (+ 819us) server_negotiation.cc:407] Connection header received
0928 16:38:24.899878 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:24.899882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:24.899929 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:24.900010 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:24.901192 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:24.901715 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:24.902786 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:24.902997 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:24.903088 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:24.905751 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:24.905755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:24.905758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:24.905784 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:24.907416 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:24.908054 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:24.908057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:24.908058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:24.908132 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:24.908527 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:24.908529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:24.908530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:24.908688 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:24.908763 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:24.909381 (+ 618us) server_negotiation.cc:299] Negotiation successful
0928 16:38:24.909519 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 16:38:24.910712 14388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:24.898050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:33803 (local address 127.27.122.194:45427)
0928 16:38:24.898467 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:24.898508 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:24.899688 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:24.900165 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:24.900171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:24.900486 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:24.901011 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:24.901032 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:24.901817 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:24.901820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:24.902680 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:24.902689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:24.903140 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:24.903142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:24.903275 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:24.903846 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:24.903861 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:24.905578 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:24.907538 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:24.907543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:24.907553 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:24.907951 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:24.908237 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:24.908242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:24.908245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:24.908428 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:24.908791 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:24.908799 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:24.909135 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 16:38:24.909327 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 16:38:24.909652 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":115,"threads_started":1}
I20260928 16:38:24.911885 14386 heartbeater.cc:347] Connected to a master server at 127.27.122.254:33803
I20260928 16:38:24.912144 14386 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:24.912700 14386 heartbeater.cc:511] Master 127.27.122.254:33803 requested a full tablet report, sending...
I20260928 16:38:24.913810 14055 ts_manager.cc:194] Registered new tserver with Master: 67f8e8863e78440da20ec45170de9a9a (127.27.122.194:41409)
I20260928 16:38:24.914620 14055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:45427
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:38:24.931694 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:33803
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:25.041373 14391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:25.041617 14391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:25.041658 14391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:25.045562 14391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:25.045610 14391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:25.045665 14391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:38:25.050661 14391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:33803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:25.051550 14391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:25.052440 14391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:25.061893 14397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:25.061942 14391 server_base.cc:1034] running on GCE node
W20260928 16:38:25.061887 14396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:25.061904 14399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:25.062500 14391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:25.063083 14391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:25.064261 14391 hybrid_clock.cc:648] HybridClock initialized: now 1790613505064232 us; error 41 us; skew 500 ppm
Sep 28 16:38:25 dist-test-slave-g9sh krb5kdc[14009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:25.066998 14391 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:38:25.067847 14391 webserver.cc:460] Webserver started at http://127.27.122.195:41941/ using document root <none> and password file <none>
I20260928 16:38:25.068392 14391 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:25.068483 14391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:25.068727 14391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:25.070758 14391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "ba75445c6bd24870ac79e086a064ec94"
format_stamp: "Formatted at 2026-09-28 16:38:25 on dist-test-slave-g9sh"
server_key: "09b234e92cb8caaf397d82f0989fad3d"
server_key_iv: "c1c163866a090653021c9afec314dc34"
server_key_version: "encryptionkey@0"
I20260928 16:38:25.071306 14391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "ba75445c6bd24870ac79e086a064ec94"
format_stamp: "Formatted at 2026-09-28 16:38:25 on dist-test-slave-g9sh"
server_key: "09b234e92cb8caaf397d82f0989fad3d"
server_key_iv: "c1c163866a090653021c9afec314dc34"
server_key_version: "encryptionkey@0"
I20260928 16:38:25.074852 14391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 16:38:25.077373 14406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:25.078621 14391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:25.078764 14391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "ba75445c6bd24870ac79e086a064ec94"
format_stamp: "Formatted at 2026-09-28 16:38:25 on dist-test-slave-g9sh"
server_key: "09b234e92cb8caaf397d82f0989fad3d"
server_key_iv: "c1c163866a090653021c9afec314dc34"
server_key_version: "encryptionkey@0"
I20260928 16:38:25.078877 14391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:25.106758 14391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:25.110064 14391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:25.110258 14391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:25.110836 14391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:25.111694 14391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:25.111763 14391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:25.111850 14391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:25.111900 14391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:25.120821 14391 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:43509
I20260928 16:38:25.120849 14520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:43509 every 8 connection(s)
I20260928 16:38:25.121742 14391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:38:25.127333 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 14391
I20260928 16:38:25.127482 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:38:25.127866 28139 external_mini_cluster.cc:1428] Setting key 23981ec30692e0851357a8dab2b58717
Sep 28 16:38:25 dist-test-slave-g9sh krb5kdc[14009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613505, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:25.135847 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.123516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:35797 (local address 127.27.122.254:33803)
0928 16:38:25.123659 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:25.123663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:25.124479 (+ 816us) server_negotiation.cc:407] Connection header received
0928 16:38:25.125950 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:25.125954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:25.126015 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:25.126120 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:25.127459 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.128153 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.129624 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.129952 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.130087 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.132520 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:25.132524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:25.132526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:25.132552 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:25.134024 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:25.134514 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:25.134517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:25.134518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:25.134619 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:25.134918 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:25.134920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:25.134921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:25.135090 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:25.135167 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:25.135626 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 16:38:25.135728 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:38:25.136620 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.123850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:33803 (local address 127.27.122.195:35797)
0928 16:38:25.124325 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:25.124367 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:25.125693 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:25.126289 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:25.126296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:25.126624 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:25.127271 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.127294 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.128269 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.128273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.129479 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.129489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.130128 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.130132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.130337 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.130933 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:25.130949 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:25.132351 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:25.134147 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:25.134152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:25.134165 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:25.134417 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:25.134705 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:25.134708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:25.134709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:25.134840 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:25.135179 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:25.135184 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:25.135398 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 16:38:25.135555 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 16:38:25.135797 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":123,"threads_started":1}
I20260928 16:38:25.137816 14521 heartbeater.cc:347] Connected to a master server at 127.27.122.254:33803
I20260928 16:38:25.138026 14521 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:25.138460 14521 heartbeater.cc:511] Master 127.27.122.254:33803 requested a full tablet report, sending...
I20260928 16:38:25.139432 14055 ts_manager.cc:194] Registered new tserver with Master: ba75445c6bd24870ac79e086a064ec94 (127.27.122.195:43509)
I20260928 16:38:25.140007 14055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:35797
I20260928 16:38:25.141908 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:38:25.152160 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.143863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.27.122.254:33803)
0928 16:38:25.144132 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:25.144136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:25.144151 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:38:25.144260 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:25.144262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:25.144297 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:25.144363 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:25.145101 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.146053 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.146960 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.147244 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.147333 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.148757 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:25.148760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:25.148762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:25.148783 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:25.150516 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:25.150987 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:25.150989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:25.150990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:25.151069 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:25.151386 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:25.151389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:25.151390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:25.151586 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:25.151706 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:25.151898 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 16:38:25.151970 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260928 16:38:25.154978 14055 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:38:25.158206 14055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:38:25.172655 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.166495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:41503 (local address 127.0.0.1:36928)
0928 16:38:25.166845 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:25.166894 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:25.167012 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:25.168058 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:25.168064 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:25.168086 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:25.168232 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.168239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.169743 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.169747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.171023 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.171032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.172115 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.172119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.172225 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.172270 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 16:38:25.172381 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 16:38:25.172497 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":85,"threads_started":1}
I20260928 16:38:25.172873 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.167295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:41409 (local address 127.0.0.1:45628)
0928 16:38:25.167959 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:25.167974 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:25.168058 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:25.168780 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:25.168782 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:25.168800 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:25.168918 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.168926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.170076 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.170079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.171254 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.171266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.172340 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.172343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.172435 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.172458 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 16:38:25.172642 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 16:38:25.172722 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"mutex_wait_us":16,"thread_start_us":97,"threads_started":1}
I20260928 16:38:25.173345 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.167736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.27.122.194:41409)
0928 16:38:25.168309 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:25.168323 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:25.168344 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:38:25.168443 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:25.168448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:25.168636 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:25.168802 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:25.169029 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.169785 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.171389 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.172238 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.172354 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.172483 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:25.173086 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 16:38:25.173203 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":139,"threads_started":1}
I20260928 16:38:25.173345 14536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.166697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:43509 (local address 127.0.0.1:43598)
0928 16:38:25.167873 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:25.167888 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:25.167983 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:25.168453 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:25.168456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:25.168474 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:25.168564 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.168570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.169953 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.169956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.171617 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.171625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.172655 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.172658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.173050 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.173074 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 16:38:25.173160 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 16:38:25.173223 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":40,"thread_start_us":99,"threads_started":1}
I20260928 16:38:25.173467 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.167305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.27.122.195:43509)
0928 16:38:25.167724 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:25.167735 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:25.168092 (+ 357us) server_negotiation.cc:407] Connection header received
0928 16:38:25.168155 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:25.168159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:25.168315 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:25.168463 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:25.168686 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.169799 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.171736 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.172490 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.172613 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.172742 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:25.173214 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 16:38:25.173316 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":134,"threads_started":1}
I20260928 16:38:25.173938 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.166631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.27.122.193:41503)
0928 16:38:25.167355 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:25.167372 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:25.167398 (+ 26us) server_negotiation.cc:407] Connection header received
0928 16:38:25.167475 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:25.167480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:25.167693 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:25.167850 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:25.168357 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.169598 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.171142 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.171959 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.172094 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.172230 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:25.173693 (+ 1463us) server_negotiation.cc:299] Negotiation successful
0928 16:38:25.173810 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":78,"threads_started":1}
I20260928 16:38:25.176371 14181 tablet_service.cc:1467] Processing CreateTablet for tablet 24f0a0ae98624efcad5951068f91fc26 (DEFAULT_TABLE table=test-table [id=bc2d8022e85b4aaf971f97d85b3557df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:25.176462 14318 tablet_service.cc:1467] Processing CreateTablet for tablet 24f0a0ae98624efcad5951068f91fc26 (DEFAULT_TABLE table=test-table [id=bc2d8022e85b4aaf971f97d85b3557df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:25.176476 14455 tablet_service.cc:1467] Processing CreateTablet for tablet 24f0a0ae98624efcad5951068f91fc26 (DEFAULT_TABLE table=test-table [id=bc2d8022e85b4aaf971f97d85b3557df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:25.177502 14455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24f0a0ae98624efcad5951068f91fc26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:25.177500 14181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24f0a0ae98624efcad5951068f91fc26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:25.177510 14318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24f0a0ae98624efcad5951068f91fc26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:25.183976 14539 tablet_bootstrap.cc:492] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836: Bootstrap starting.
I20260928 16:38:25.185204 14540 tablet_bootstrap.cc:492] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94: Bootstrap starting.
I20260928 16:38:25.186179 14539 tablet_bootstrap.cc:654] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:25.186940 14540 tablet_bootstrap.cc:654] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:25.187320 14539 log.cc:824] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:25.188130 14540 log.cc:824] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:25.188758 14541 tablet_bootstrap.cc:492] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a: Bootstrap starting.
I20260928 16:38:25.189635 14539 tablet_bootstrap.cc:492] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836: No bootstrap required, opened a new log
I20260928 16:38:25.189833 14539 ts_tablet_manager.cc:1397] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260928 16:38:25.190058 14540 tablet_bootstrap.cc:492] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94: No bootstrap required, opened a new log
I20260928 16:38:25.190250 14540 ts_tablet_manager.cc:1397] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 16:38:25.190572 14541 tablet_bootstrap.cc:654] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:25.191556 14541 log.cc:824] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:25.193234 14541 tablet_bootstrap.cc:492] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a: No bootstrap required, opened a new log
I20260928 16:38:25.193419 14541 ts_tablet_manager.cc:1397] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:38:25.193837 14539 raft_consensus.cc:348] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4114afe5ea9f4310933143efc68a1836" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41503 } } peers { permanent_uuid: "ba75445c6bd24870ac79e086a064ec94" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 43509 } } peers { permanent_uuid: "67f8e8863e78440da20ec45170de9a9a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41409 } }
I20260928 16:38:25.193908 14540 raft_consensus.cc:348] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4114afe5ea9f4310933143efc68a1836" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41503 } } peers { permanent_uuid: "ba75445c6bd24870ac79e086a064ec94" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 43509 } } peers { permanent_uuid: "67f8e8863e78440da20ec45170de9a9a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41409 } }
I20260928 16:38:25.194113 14540 raft_consensus.cc:374] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:25.194114 14539 raft_consensus.cc:374] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:25.194172 14539 raft_consensus.cc:729] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4114afe5ea9f4310933143efc68a1836, State: Initialized, Role: FOLLOWER
I20260928 16:38:25.194199 14540 raft_consensus.cc:729] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba75445c6bd24870ac79e086a064ec94, State: Initialized, Role: FOLLOWER
I20260928 16:38:25.194670 14540 consensus_queue.cc:260] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4114afe5ea9f4310933143efc68a1836" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41503 } } peers { permanent_uuid: "ba75445c6bd24870ac79e086a064ec94" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 43509 } } peers { permanent_uuid: "67f8e8863e78440da20ec45170de9a9a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41409 } }
I20260928 16:38:25.194670 14539 consensus_queue.cc:260] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4114afe5ea9f4310933143efc68a1836" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41503 } } peers { permanent_uuid: "ba75445c6bd24870ac79e086a064ec94" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 43509 } } peers { permanent_uuid: "67f8e8863e78440da20ec45170de9a9a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41409 } }
I20260928 16:38:25.195634 14540 ts_tablet_manager.cc:1428] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 16:38:25.195634 14539 ts_tablet_manager.cc:1428] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:38:25.196022 14250 heartbeater.cc:503] Master 127.27.122.254:33803 was elected leader, sending a full tablet report...
I20260928 16:38:25.197190 14521 heartbeater.cc:503] Master 127.27.122.254:33803 was elected leader, sending a full tablet report...
I20260928 16:38:25.197268 14541 raft_consensus.cc:348] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4114afe5ea9f4310933143efc68a1836" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41503 } } peers { permanent_uuid: "ba75445c6bd24870ac79e086a064ec94" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 43509 } } peers { permanent_uuid: "67f8e8863e78440da20ec45170de9a9a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41409 } }
I20260928 16:38:25.197525 14541 raft_consensus.cc:374] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:25.197589 14541 raft_consensus.cc:729] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67f8e8863e78440da20ec45170de9a9a, State: Initialized, Role: FOLLOWER
I20260928 16:38:25.198009 14541 consensus_queue.cc:260] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4114afe5ea9f4310933143efc68a1836" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41503 } } peers { permanent_uuid: "ba75445c6bd24870ac79e086a064ec94" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 43509 } } peers { permanent_uuid: "67f8e8863e78440da20ec45170de9a9a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41409 } }
I20260928 16:38:25.198722 14386 heartbeater.cc:503] Master 127.27.122.254:33803 was elected leader, sending a full tablet report...
I20260928 16:38:25.199060 14541 ts_tablet_manager.cc:1428] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 16:38:25.242415 14548 raft_consensus.cc:482] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:38:25.242611 14548 raft_consensus.cc:504] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4114afe5ea9f4310933143efc68a1836" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41503 } } peers { permanent_uuid: "ba75445c6bd24870ac79e086a064ec94" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 43509 } } peers { permanent_uuid: "67f8e8863e78440da20ec45170de9a9a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41409 } }
I20260928 16:38:25.243815 14548 leader_election.cc:288] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4114afe5ea9f4310933143efc68a1836 (127.27.122.193:41503), ba75445c6bd24870ac79e086a064ec94 (127.27.122.195:43509)
I20260928 16:38:25.247745 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.244167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:52007 (local address 127.27.122.195:43509)
0928 16:38:25.244293 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:25.244299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:25.244620 (+ 321us) server_negotiation.cc:407] Connection header received
0928 16:38:25.244700 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:25.244703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:25.244753 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:25.244833 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:25.245141 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.245654 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.246759 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.247167 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.247261 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.247342 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:25.247533 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 16:38:25.247610 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 16:38:25.247764 14388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.244040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:41503 (local address 127.27.122.194:51917)
0928 16:38:25.244244 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:25.244259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:25.244355 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:25.244665 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:25.244668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:25.244688 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:25.244828 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.244835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.245659 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.245663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.246636 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.246646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.247358 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.247363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.247451 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.247468 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 16:38:25.247572 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 16:38:25.247640 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":62}
I20260928 16:38:25.247782 14549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.244083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:43509 (local address 127.27.122.194:52007)
0928 16:38:25.244500 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:25.244514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:25.244617 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:25.244851 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:25.244853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:25.244867 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:25.245035 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.245042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.245759 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.245762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.246632 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.246641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.247288 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.247294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.247391 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.247417 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 16:38:25.247500 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 16:38:25.247582 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":37,"thread_start_us":131,"threads_started":1}
I20260928 16:38:25.247797 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.244161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:51917 (local address 127.27.122.193:41503)
0928 16:38:25.244292 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:25.244296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:25.244313 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:38:25.244481 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:25.244485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:25.244554 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:25.244636 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:25.244958 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.245542 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.246797 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.247204 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.247352 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.247405 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:25.247608 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 16:38:25.247688 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:38:25.248428 14202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f0a0ae98624efcad5951068f91fc26" candidate_uuid: "67f8e8863e78440da20ec45170de9a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4114afe5ea9f4310933143efc68a1836" is_pre_election: true
I20260928 16:38:25.248517 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f0a0ae98624efcad5951068f91fc26" candidate_uuid: "67f8e8863e78440da20ec45170de9a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba75445c6bd24870ac79e086a064ec94" is_pre_election: true
I20260928 16:38:25.248768 14475 raft_consensus.cc:2454] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67f8e8863e78440da20ec45170de9a9a in term 0.
I20260928 16:38:25.248768 14202 raft_consensus.cc:2454] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67f8e8863e78440da20ec45170de9a9a in term 0.
I20260928 16:38:25.249290 14272 leader_election.cc:302] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67f8e8863e78440da20ec45170de9a9a, ba75445c6bd24870ac79e086a064ec94; no voters:
I20260928 16:38:25.249568 14548 raft_consensus.cc:2781] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:38:25.249653 14548 raft_consensus.cc:482] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:38:25.249696 14548 raft_consensus.cc:3037] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:25.250717 14548 raft_consensus.cc:504] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4114afe5ea9f4310933143efc68a1836" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41503 } } peers { permanent_uuid: "ba75445c6bd24870ac79e086a064ec94" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 43509 } } peers { permanent_uuid: "67f8e8863e78440da20ec45170de9a9a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41409 } }
I20260928 16:38:25.251044 14548 leader_election.cc:288] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [CANDIDATE]: Term 1 election: Requested vote from peers 4114afe5ea9f4310933143efc68a1836 (127.27.122.193:41503), ba75445c6bd24870ac79e086a064ec94 (127.27.122.195:43509)
I20260928 16:38:25.251379 14475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f0a0ae98624efcad5951068f91fc26" candidate_uuid: "67f8e8863e78440da20ec45170de9a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba75445c6bd24870ac79e086a064ec94"
I20260928 16:38:25.251454 14202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24f0a0ae98624efcad5951068f91fc26" candidate_uuid: "67f8e8863e78440da20ec45170de9a9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4114afe5ea9f4310933143efc68a1836"
I20260928 16:38:25.251477 14475 raft_consensus.cc:3037] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:25.251624 14202 raft_consensus.cc:3037] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:25.252573 14475 raft_consensus.cc:2454] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67f8e8863e78440da20ec45170de9a9a in term 1.
I20260928 16:38:25.252717 14202 raft_consensus.cc:2454] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67f8e8863e78440da20ec45170de9a9a in term 1.
I20260928 16:38:25.252866 14272 leader_election.cc:302] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67f8e8863e78440da20ec45170de9a9a, ba75445c6bd24870ac79e086a064ec94; no voters:
I20260928 16:38:25.253121 14548 raft_consensus.cc:2781] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:25.253346 14548 raft_consensus.cc:686] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [term 1 LEADER]: Becoming Leader. State: Replica: 67f8e8863e78440da20ec45170de9a9a, State: Running, Role: LEADER
I20260928 16:38:25.253679 14548 consensus_queue.cc:237] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4114afe5ea9f4310933143efc68a1836" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41503 } } peers { permanent_uuid: "ba75445c6bd24870ac79e086a064ec94" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 43509 } } peers { permanent_uuid: "67f8e8863e78440da20ec45170de9a9a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41409 } }
I20260928 16:38:25.256708 14055 catalog_manager.cc:5495] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 67f8e8863e78440da20ec45170de9a9a (127.27.122.194). New cstate: current_term: 1 leader_uuid: "67f8e8863e78440da20ec45170de9a9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4114afe5ea9f4310933143efc68a1836" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba75445c6bd24870ac79e086a064ec94" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 43509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67f8e8863e78440da20ec45170de9a9a" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 41409 } health_report { overall_health: HEALTHY } } }
I20260928 16:38:25.268775 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.265583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45632 (local address 127.27.122.194:41409)
0928 16:38:25.265698 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:25.265702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:25.265713 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:38:25.265833 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:25.265835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:25.265884 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:25.265967 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:38:25.266187 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.266678 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.267500 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.267718 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.267789 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.268083 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:38:25.268449 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:38:25.268573 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:25.268646 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 16:38:25.268677 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:38:25.274245 14202 raft_consensus.cc:1261] T 24f0a0ae98624efcad5951068f91fc26 P 4114afe5ea9f4310933143efc68a1836 [term 1 FOLLOWER]: Refusing update from remote peer 67f8e8863e78440da20ec45170de9a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:25.274245 14475 raft_consensus.cc:1261] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94 [term 1 FOLLOWER]: Refusing update from remote peer 67f8e8863e78440da20ec45170de9a9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:25.274931 14550 consensus_queue.cc:1035] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba75445c6bd24870ac79e086a064ec94" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 43509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:38:25.275117 14548 consensus_queue.cc:1035] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4114afe5ea9f4310933143efc68a1836" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 41503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:38:25.283313 14555 mvcc.cc:205] Tried to move back new op lower bound from 7334352917596479488 to 7334352917521879040. Current Snapshot: MvccSnapshot[applied={T|T < 7334352917596479488}]
I20260928 16:38:25.284920 14556 mvcc.cc:205] Tried to move back new op lower bound from 7334352917596479488 to 7334352917521879040. Current Snapshot: MvccSnapshot[applied={T|T < 7334352917596479488 or (T in {7334352917596479488})}]
I20260928 16:38:25.292034 14054 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:38:25.292182 14054 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:38:25.294584 14054 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b11bc9556d764c34843543de249b6431: Sending DeleteTablet for 3 replicas of tablet 24f0a0ae98624efcad5951068f91fc26
I20260928 16:38:25.295374 14318 tablet_service.cc:1514] Processing DeleteTablet for tablet 24f0a0ae98624efcad5951068f91fc26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:25 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:45628
I20260928 16:38:25.295374 14455 tablet_service.cc:1514] Processing DeleteTablet for tablet 24f0a0ae98624efcad5951068f91fc26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:25 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:43598
I20260928 16:38:25.295504 14181 tablet_service.cc:1514] Processing DeleteTablet for tablet 24f0a0ae98624efcad5951068f91fc26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:25 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:36928
I20260928 16:38:25.295847 14565 tablet_replica.cc:330] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94: stopping tablet replica
I20260928 16:38:25.296003 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 14118
I20260928 16:38:25.296157 14564 tablet_replica.cc:330] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a: stopping tablet replica
I20260928 16:38:25.296435 14564 raft_consensus.cc:2229] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:38:25.296527 14565 raft_consensus.cc:2229] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:38:25.296857 14565 raft_consensus.cc:2258] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:25.296864 14564 raft_consensus.cc:2258] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:25.298379 14564 ts_tablet_manager.cc:1905] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:38:25.298382 14565 ts_tablet_manager.cc:1905] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:38:25.301251 14564 ts_tablet_manager.cc:1918] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:38:25.301353 14564 log.cc:1196] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/24f0a0ae98624efcad5951068f91fc26
I20260928 16:38:25.301710 14564 ts_tablet_manager.cc:1939] T 24f0a0ae98624efcad5951068f91fc26 P 67f8e8863e78440da20ec45170de9a9a: Deleting consensus metadata
I20260928 16:38:25.301872 14565 ts_tablet_manager.cc:1918] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:38:25.301975 14565 log.cc:1196] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/24f0a0ae98624efcad5951068f91fc26
I20260928 16:38:25.302315 14565 ts_tablet_manager.cc:1939] T 24f0a0ae98624efcad5951068f91fc26 P ba75445c6bd24870ac79e086a064ec94: Deleting consensus metadata
I20260928 16:38:25.302799 14041 catalog_manager.cc:4841] TS 67f8e8863e78440da20ec45170de9a9a (127.27.122.194:41409): tablet 24f0a0ae98624efcad5951068f91fc26 (table test-table [id=bc2d8022e85b4aaf971f97d85b3557df]) successfully deleted
I20260928 16:38:25.303436 14039 catalog_manager.cc:4841] TS ba75445c6bd24870ac79e086a064ec94 (127.27.122.195:43509): tablet 24f0a0ae98624efcad5951068f91fc26 (table test-table [id=bc2d8022e85b4aaf971f97d85b3557df]) successfully deleted
W20260928 16:38:25.304951 14038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:41503 (error 108)
I20260928 16:38:25.305940 14536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.305650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:41503 (local address 127.0.0.1:36944)
0928 16:38:25.305775 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:25.305847 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:41503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 16:38:25.306039 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 14255
W20260928 16:38:25.306171 14038 catalog_manager.cc:4584] TS 4114afe5ea9f4310933143efc68a1836 (127.27.122.193:41503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 24f0a0ae98624efcad5951068f91fc26: Network error: Client connection negotiation failed: client connection to 127.27.122.193:41503: connect: Connection refused (error 111)
I20260928 16:38:25.311909 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 14391
I20260928 16:38:25.317440 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 14022
2026-09-28T16:38:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2290 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T16:38:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:25Z Disabled control of system clock
I20260928 16:38:25.347681 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:40473
--webserver_interface=127.27.122.254
--webserver_port=0
--builtin_ntp_servers=127.27.122.212:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:40473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:38:25.453173 14571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:25.453397 14571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:25.453452 14571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:25.457317 14571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:25.457362 14571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:25.457377 14571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:25.457393 14571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:25.457444 14571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:25.462076 14571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:40473
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:40473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:25.462994 14571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:25.464236 14571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:25.473090 14577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:25.473114 14576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:25.473217 14579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:25.474274 14571 server_base.cc:1034] running on GCE node
I20260928 16:38:25.474740 14571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:25.475706 14571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:25.476931 14571 hybrid_clock.cc:648] HybridClock initialized: now 1790613505476906 us; error 33 us; skew 500 ppm
I20260928 16:38:25.478664 14571 webserver.cc:460] Webserver started at http://127.27.122.254:46633/ using document root <none> and password file <none>
I20260928 16:38:25.479228 14571 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:25.479321 14571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:25.479516 14571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:25.481495 14571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "ecf3ebb2b3b14aaf875963550da17d87"
format_stamp: "Formatted at 2026-09-28 16:38:25 on dist-test-slave-g9sh"
server_key: "d07f60ae50c62e863069a1ed210a22df"
server_key_iv: "a752b9622412b2ade7530a3c9048fafd"
server_key_version: "encryptionkey@0"
I20260928 16:38:25.482035 14571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "ecf3ebb2b3b14aaf875963550da17d87"
format_stamp: "Formatted at 2026-09-28 16:38:25 on dist-test-slave-g9sh"
server_key: "d07f60ae50c62e863069a1ed210a22df"
server_key_iv: "a752b9622412b2ade7530a3c9048fafd"
server_key_version: "encryptionkey@0"
I20260928 16:38:25.485867 14571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 16:38:25.488163 14585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:25.489379 14571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:25.489512 14571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "ecf3ebb2b3b14aaf875963550da17d87"
format_stamp: "Formatted at 2026-09-28 16:38:25 on dist-test-slave-g9sh"
server_key: "d07f60ae50c62e863069a1ed210a22df"
server_key_iv: "a752b9622412b2ade7530a3c9048fafd"
server_key_version: "encryptionkey@0"
I20260928 16:38:25.489619 14571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:25.507342 14571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:25.507942 14571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:25.508128 14571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:25.514954 14571 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:40473
I20260928 16:38:25.514986 14638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:40473 every 8 connection(s)
I20260928 16:38:25.515837 14571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:25.518797 14639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:25.522807 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 14571
I20260928 16:38:25.522903 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:25.523200 28139 external_mini_cluster.cc:1428] Setting key fa554a847aec04ac1a438bc70b2008f5
I20260928 16:38:25.524391 14639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87: Bootstrap starting.
I20260928 16:38:25.526615 14639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:25.527338 14639 log.cc:824] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:25.529115 14639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87: No bootstrap required, opened a new log
I20260928 16:38:25.532060 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.524498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.27.122.254:40473)
0928 16:38:25.525146 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:25.525161 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:25.525195 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:38:25.525795 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:25.525810 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:25.526182 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:25.526506 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:25.526929 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.527916 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.528815 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.529398 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.529571 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.530431 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:25.530435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:25.530447 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:25.530811 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:25.530867 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:25.530893 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:25.531010 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:25.531158 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 16:38:25.531392 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
I20260928 16:38:25.532764 14639 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf3ebb2b3b14aaf875963550da17d87" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40473 } }
I20260928 16:38:25.533023 14639 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:25.533090 14639 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf3ebb2b3b14aaf875963550da17d87, State: Initialized, Role: FOLLOWER
I20260928 16:38:25.533506 14639 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf3ebb2b3b14aaf875963550da17d87" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40473 } }
I20260928 16:38:25.533627 14639 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:25.533711 14639 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:25.533816 14639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:25.534765 14639 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf3ebb2b3b14aaf875963550da17d87" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40473 } }
I20260928 16:38:25.535097 14639 leader_election.cc:302] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecf3ebb2b3b14aaf875963550da17d87; no voters:
I20260928 16:38:25.535405 14639 leader_election.cc:288] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:25.535528 14644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:25.535753 14644 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [term 1 LEADER]: Becoming Leader. State: Replica: ecf3ebb2b3b14aaf875963550da17d87, State: Running, Role: LEADER
I20260928 16:38:25.536052 14644 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf3ebb2b3b14aaf875963550da17d87" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40473 } }
I20260928 16:38:25.536620 14639 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:25.537688 14645 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecf3ebb2b3b14aaf875963550da17d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf3ebb2b3b14aaf875963550da17d87" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40473 } } }
I20260928 16:38:25.537676 14646 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecf3ebb2b3b14aaf875963550da17d87. Latest consensus state: current_term: 1 leader_uuid: "ecf3ebb2b3b14aaf875963550da17d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf3ebb2b3b14aaf875963550da17d87" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40473 } } }
I20260928 16:38:25.537822 14646 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:25.537822 14645 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:25.538221 14653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:25.541226 14653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:25.546409 14653 catalog_manager.cc:1347] Generated new cluster ID: d8de35e5324c4d1bafebd69087d78308
I20260928 16:38:25.546478 14653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:25.557252 14653 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:25.558091 14653 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:25.565246 14653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87: Generated new TSK 0
I20260928 16:38:25.565871 14653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:25.570451 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:40473
--builtin_ntp_servers=127.27.122.212:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:38:25.677978 14664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:25.678236 14664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:25.678332 14664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:25.682330 14664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:25.682399 14664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:25.682482 14664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:38:25.687127 14664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:40473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:25.688124 14664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:25.689443 14664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:25.698904 14671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:25.698889 14669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:25.698869 14673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:25.699150 14664 server_base.cc:1034] running on GCE node
I20260928 16:38:25.699591 14664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:25.700193 14664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:25.701380 14664 hybrid_clock.cc:648] HybridClock initialized: now 1790613505701348 us; error 42 us; skew 500 ppm
I20260928 16:38:25.703215 14664 webserver.cc:460] Webserver started at http://127.27.122.193:41095/ using document root <none> and password file <none>
I20260928 16:38:25.703770 14664 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:25.703858 14664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:25.704080 14664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:25.706049 14664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "0a24e0afa6dc4da886897c9447dba506"
format_stamp: "Formatted at 2026-09-28 16:38:25 on dist-test-slave-g9sh"
server_key: "4a7c5142f22bcddd858a93bfcea24f94"
server_key_iv: "8a0732e0fd1878e25a42678ad598acd1"
server_key_version: "encryptionkey@0"
I20260928 16:38:25.706617 14664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "0a24e0afa6dc4da886897c9447dba506"
format_stamp: "Formatted at 2026-09-28 16:38:25 on dist-test-slave-g9sh"
server_key: "4a7c5142f22bcddd858a93bfcea24f94"
server_key_iv: "8a0732e0fd1878e25a42678ad598acd1"
server_key_version: "encryptionkey@0"
I20260928 16:38:25.710089 14664 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:38:25.712350 14679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:25.713621 14664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:38:25.713753 14664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "0a24e0afa6dc4da886897c9447dba506"
format_stamp: "Formatted at 2026-09-28 16:38:25 on dist-test-slave-g9sh"
server_key: "4a7c5142f22bcddd858a93bfcea24f94"
server_key_iv: "8a0732e0fd1878e25a42678ad598acd1"
server_key_version: "encryptionkey@0"
I20260928 16:38:25.713856 14664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:25.728099 14664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:25.728675 14664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:25.728858 14664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:25.729394 14664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:25.730247 14664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:25.730321 14664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:25.730391 14664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:25.730436 14664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:25.739490 14664 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:39523
I20260928 16:38:25.739527 14795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:39523 every 8 connection(s)
I20260928 16:38:25.740300 14664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:38:25.746593 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 14664
I20260928 16:38:25.746721 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:38:25.747020 28139 external_mini_cluster.cc:1428] Setting key 60567b68d801e7f7afa0b995e48865be
I20260928 16:38:25.748898 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:40473
--builtin_ntp_servers=127.27.122.212:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:38:25.749781 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.742063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:51947 (local address 127.27.122.254:40473)
0928 16:38:25.742219 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:25.742223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:25.743176 (+ 953us) server_negotiation.cc:407] Connection header received
0928 16:38:25.744533 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:25.744536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:25.744584 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:25.744657 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:25.745617 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.746140 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.747442 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.747807 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.747934 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.749251 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:25.749255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:25.749257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:25.749269 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:25.749292 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:25.749301 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:25.749343 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:25.749562 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 16:38:25.749629 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 16:38:25.750854 14798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.742387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:40473 (local address 127.27.122.193:51947)
0928 16:38:25.743014 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:25.743066 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:25.744324 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:25.744817 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:25.744823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:25.744974 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:25.745481 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.745495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.746256 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.746259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.747304 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.747316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.747974 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.747979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.748166 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.748843 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:38:25.748845 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:25.748882 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:38:25.749031 (+ 149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:25.749403 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:25.749411 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:25.749426 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 16:38:25.749555 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 16:38:25.749814 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260928 16:38:25.752133 14796 heartbeater.cc:347] Connected to a master server at 127.27.122.254:40473
I20260928 16:38:25.752346 14796 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:25.752835 14796 heartbeater.cc:511] Master 127.27.122.254:40473 requested a full tablet report, sending...
I20260928 16:38:25.754293 14602 ts_manager.cc:194] Registered new tserver with Master: 0a24e0afa6dc4da886897c9447dba506 (127.27.122.193:39523)
I20260928 16:38:25.755398 14602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.122.193:51947
W20260928 16:38:25.859050 14799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:25.859341 14799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:25.859421 14799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:25.863395 14799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:25.863468 14799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:25.863562 14799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:38:25.868098 14799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:40473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:25.869097 14799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:25.870368 14799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:25.879454 14804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:25.879483 14807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:25.879673 14805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:25.879853 14799 server_base.cc:1034] running on GCE node
I20260928 16:38:25.880256 14799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:25.880826 14799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:25.882021 14799 hybrid_clock.cc:648] HybridClock initialized: now 1790613505881990 us; error 40 us; skew 500 ppm
I20260928 16:38:25.883709 14799 webserver.cc:460] Webserver started at http://127.27.122.194:42673/ using document root <none> and password file <none>
I20260928 16:38:25.884243 14799 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:25.884330 14799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:25.884552 14799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:25.886500 14799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "84dc74c19043467a9fa5a9399bcee4b5"
format_stamp: "Formatted at 2026-09-28 16:38:25 on dist-test-slave-g9sh"
server_key: "249f11d643e07945f469c3311a30c5bc"
server_key_iv: "4ab5500c6b0e022f879e47508d9f3fa0"
server_key_version: "encryptionkey@0"
I20260928 16:38:25.887038 14799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "84dc74c19043467a9fa5a9399bcee4b5"
format_stamp: "Formatted at 2026-09-28 16:38:25 on dist-test-slave-g9sh"
server_key: "249f11d643e07945f469c3311a30c5bc"
server_key_iv: "4ab5500c6b0e022f879e47508d9f3fa0"
server_key_version: "encryptionkey@0"
I20260928 16:38:25.890498 14799 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:38:25.892753 14813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:25.894002 14799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:38:25.894126 14799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "84dc74c19043467a9fa5a9399bcee4b5"
format_stamp: "Formatted at 2026-09-28 16:38:25 on dist-test-slave-g9sh"
server_key: "249f11d643e07945f469c3311a30c5bc"
server_key_iv: "4ab5500c6b0e022f879e47508d9f3fa0"
server_key_version: "encryptionkey@0"
I20260928 16:38:25.894230 14799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:25.911594 14799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:25.912371 14799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:25.912580 14799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:25.913314 14799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:25.914515 14799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:25.914583 14799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:25.914644 14799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:25.914686 14799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:25.924391 14799 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:39501
I20260928 16:38:25.924410 14930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:39501 every 8 connection(s)
I20260928 16:38:25.925341 14799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:38:25.934051 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.927131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:54819 (local address 127.27.122.254:40473)
0928 16:38:25.927299 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:25.927305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:25.928351 (+ 1046us) server_negotiation.cc:407] Connection header received
0928 16:38:25.929762 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:25.929765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:25.929818 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:25.929901 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:25.930760 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.931285 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.932380 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.932612 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.932692 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.933657 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:25.933660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:25.933662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:25.933674 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:25.933694 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:25.933702 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:25.933768 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:25.933894 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 16:38:25.933943 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:38:25.934819 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:25.927572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:40473 (local address 127.27.122.194:54819)
0928 16:38:25.928189 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:25.928244 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:25.929560 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:25.930053 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:25.930059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:25.930136 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:25.930617 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.930629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.931411 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.931413 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.932274 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:25.932282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:25.932739 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:25.932741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:25.932869 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:25.933429 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:38:25.933430 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:25.933465 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:38:25.933528 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:25.933781 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:25.933787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:25.933791 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 16:38:25.933876 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 16:38:25.934064 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":125,"threads_started":1}
I20260928 16:38:25.935194 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 14799
I20260928 16:38:25.935367 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:38:25.935717 28139 external_mini_cluster.cc:1428] Setting key 0eb53bfc69ca536fde43e91b301aef96
I20260928 16:38:25.936017 14931 heartbeater.cc:347] Connected to a master server at 127.27.122.254:40473
I20260928 16:38:25.936265 14931 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:25.936776 14931 heartbeater.cc:511] Master 127.27.122.254:40473 requested a full tablet report, sending...
I20260928 16:38:25.937757 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:40473
--builtin_ntp_servers=127.27.122.212:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:38:25.937865 14602 ts_manager.cc:194] Registered new tserver with Master: 84dc74c19043467a9fa5a9399bcee4b5 (127.27.122.194:39501)
I20260928 16:38:25.938674 14602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.122.194:54819
W20260928 16:38:26.046517 14934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:26.046880 14934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:26.046941 14934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:26.051059 14934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:26.051105 14934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:26.051156 14934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:38:26.055757 14934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:40473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:26.056751 14934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:26.058089 14934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:26.067732 14940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:26.067741 14939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:26.067741 14942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:26.067888 14934 server_base.cc:1034] running on GCE node
I20260928 16:38:26.068462 14934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:26.069129 14934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:26.070322 14934 hybrid_clock.cc:648] HybridClock initialized: now 1790613506070294 us; error 46 us; skew 500 ppm
I20260928 16:38:26.072079 14934 webserver.cc:460] Webserver started at http://127.27.122.195:43523/ using document root <none> and password file <none>
I20260928 16:38:26.072645 14934 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:26.072736 14934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:26.073010 14934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:26.075002 14934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "cc7d7cffd8854020899b724e7c6bbc1c"
format_stamp: "Formatted at 2026-09-28 16:38:26 on dist-test-slave-g9sh"
server_key: "7b1fca567426ebca10bba70c453a922b"
server_key_iv: "d8642f2a918f6d845331a14be653b1ea"
server_key_version: "encryptionkey@0"
I20260928 16:38:26.075583 14934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "cc7d7cffd8854020899b724e7c6bbc1c"
format_stamp: "Formatted at 2026-09-28 16:38:26 on dist-test-slave-g9sh"
server_key: "7b1fca567426ebca10bba70c453a922b"
server_key_iv: "d8642f2a918f6d845331a14be653b1ea"
server_key_version: "encryptionkey@0"
I20260928 16:38:26.079103 14934 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:38:26.081507 14948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:26.082652 14934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:26.082796 14934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "cc7d7cffd8854020899b724e7c6bbc1c"
format_stamp: "Formatted at 2026-09-28 16:38:26 on dist-test-slave-g9sh"
server_key: "7b1fca567426ebca10bba70c453a922b"
server_key_iv: "d8642f2a918f6d845331a14be653b1ea"
server_key_version: "encryptionkey@0"
I20260928 16:38:26.082927 14934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:26.098199 14934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:26.098783 14934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:26.098968 14934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:26.099529 14934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:26.100389 14934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:26.100467 14934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:26.100557 14934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:26.100608 14934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:26.109612 14934 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:40763
I20260928 16:38:26.109642 15066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:40763 every 8 connection(s)
I20260928 16:38:26.110430 14934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:38:26.113911 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 14934
I20260928 16:38:26.114030 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:38:26.114308 28139 external_mini_cluster.cc:1428] Setting key 5135e07c5e0cc1e03a918d266f10b801
I20260928 16:38:26.120101 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.112216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:55839 (local address 127.27.122.254:40473)
0928 16:38:26.112351 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:26.112355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:26.113459 (+ 1104us) server_negotiation.cc:407] Connection header received
0928 16:38:26.115281 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:26.115288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:26.115345 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:26.115425 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:26.116432 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.117113 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.118323 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.118576 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.118659 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.119657 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:26.119660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:26.119662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:26.119673 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:26.119692 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:26.119699 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:26.119764 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:26.119938 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 16:38:26.119996 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:38:26.120883 15069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.112581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:40473 (local address 127.27.122.195:55839)
0928 16:38:26.113322 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:26.113360 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:26.115022 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:26.115632 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:26.115647 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:26.115749 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:26.116253 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:26.116265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.117258 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.117261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:26.118211 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:26.118221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.118708 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.118712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:26.118838 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.119413 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:38:26.119414 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:26.119465 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:38:26.119524 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:26.119801 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:26.119811 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:26.119816 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 16:38:26.119921 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 16:38:26.120126 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":139,"threads_started":1}
I20260928 16:38:26.121973 15067 heartbeater.cc:347] Connected to a master server at 127.27.122.254:40473
I20260928 16:38:26.122161 15067 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:26.122601 15067 heartbeater.cc:511] Master 127.27.122.254:40473 requested a full tablet report, sending...
I20260928 16:38:26.123507 14602 ts_manager.cc:194] Registered new tserver with Master: cc7d7cffd8854020899b724e7c6bbc1c (127.27.122.195:40763)
I20260928 16:38:26.124055 14602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.122.195:55839
I20260928 16:38:26.128178 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:38:26.133767 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.130114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.27.122.254:40473)
0928 16:38:26.130283 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:26.130286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:26.130404 (+ 118us) server_negotiation.cc:407] Connection header received
0928 16:38:26.130464 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:26.130466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:26.130502 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:26.130568 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:26.130823 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.131319 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.132133 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.132337 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.132416 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.133266 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:26.133269 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:26.133271 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:26.133282 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:26.133301 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:26.133308 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:26.133411 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:26.133570 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 16:38:26.133615 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 16:38:26.136708 14602 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:38:26.139987 14602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:38:26.154608 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.148463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:39501 (local address 127.0.0.1:49062)
0928 16:38:26.149257 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:26.149307 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:26.149442 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:26.150282 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:26.150289 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:26.150313 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:26.150471 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:26.150477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.151679 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.151686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:26.152966 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:26.152977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.154064 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.154070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:26.154180 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.154217 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 16:38:26.154327 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 16:38:26.154444 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":38,"thread_start_us":82,"threads_started":1}
I20260928 16:38:26.154660 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.148632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.27.122.194:39501)
0928 16:38:26.149753 (+ 1121us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:26.149766 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:26.149791 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:38:26.149874 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:26.149879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:26.150083 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:26.150606 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:26.150778 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.151567 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.153105 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.153922 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.154044 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.154217 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:26.154377 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 16:38:26.154479 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1000,"thread_start_us":177,"threads_started":1}
I20260928 16:38:26.155082 15081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.149163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:39523 (local address 127.0.0.1:60232)
0928 16:38:26.149690 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:26.149701 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:26.149785 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:26.150610 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:26.150616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:26.150638 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:26.150731 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:26.150737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.152189 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.152193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:26.153415 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:26.153424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.154668 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.154674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:26.154770 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.154798 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 16:38:26.154872 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 16:38:26.154943 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":65,"thread_start_us":100,"threads_started":1}
I20260928 16:38:26.155153 15082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.149322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.27.122.195:40763)
0928 16:38:26.150265 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:26.150277 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:26.150299 (+ 22us) server_negotiation.cc:407] Connection header received
0928 16:38:26.150373 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:26.150381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:26.150545 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:26.150696 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:26.150916 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.152066 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.153482 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.154040 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.154156 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.154328 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:26.154880 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 16:38:26.154981 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":96,"threads_started":1}
I20260928 16:38:26.155181 15083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.149592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.27.122.193:39523)
0928 16:38:26.149918 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:26.149933 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:26.149958 (+ 25us) server_negotiation.cc:407] Connection header received
0928 16:38:26.150037 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:26.150042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:26.150279 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:26.150442 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:26.150864 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.152071 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.153549 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.154339 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.154450 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.154600 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:26.154917 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 16:38:26.155013 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":133,"threads_started":1}
I20260928 16:38:26.154798 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.148886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:40763 (local address 127.0.0.1:35660)
0928 16:38:26.149359 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:26.149372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:26.149442 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:26.150689 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:26.150692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:26.150708 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:26.150814 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:26.150821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.152205 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.152208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:26.153362 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:26.153371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.154387 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.154390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:26.154483 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.154515 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 16:38:26.154582 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 16:38:26.154647 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":25,"thread_start_us":106,"threads_started":1}
I20260928 16:38:26.158051 14862 tablet_service.cc:1467] Processing CreateTablet for tablet 9712d90645914cd8b2d4c9fb909f9ff9 (DEFAULT_TABLE table=test-table [id=84dda325c0fa4b088b5a4b3f880f6cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:26.158149 14729 tablet_service.cc:1467] Processing CreateTablet for tablet 9712d90645914cd8b2d4c9fb909f9ff9 (DEFAULT_TABLE table=test-table [id=84dda325c0fa4b088b5a4b3f880f6cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:26.159041 14862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9712d90645914cd8b2d4c9fb909f9ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:26.159150 14729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9712d90645914cd8b2d4c9fb909f9ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:26.159010 14999 tablet_service.cc:1467] Processing CreateTablet for tablet 9712d90645914cd8b2d4c9fb909f9ff9 (DEFAULT_TABLE table=test-table [id=84dda325c0fa4b088b5a4b3f880f6cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:26.159921 14999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9712d90645914cd8b2d4c9fb909f9ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:26.165067 15084 tablet_bootstrap.cc:492] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5: Bootstrap starting.
I20260928 16:38:26.166671 15085 tablet_bootstrap.cc:492] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506: Bootstrap starting.
I20260928 16:38:26.167232 15086 tablet_bootstrap.cc:492] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c: Bootstrap starting.
I20260928 16:38:26.167711 15084 tablet_bootstrap.cc:654] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:26.168496 15085 tablet_bootstrap.cc:654] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:26.168613 15084 log.cc:824] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:26.169298 15086 tablet_bootstrap.cc:654] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:26.169389 15085 log.cc:824] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:26.170099 15086 log.cc:824] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:26.170722 15084 tablet_bootstrap.cc:492] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5: No bootstrap required, opened a new log
I20260928 16:38:26.170934 15084 ts_tablet_manager.cc:1397] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260928 16:38:26.171268 15085 tablet_bootstrap.cc:492] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506: No bootstrap required, opened a new log
I20260928 16:38:26.171511 15085 ts_tablet_manager.cc:1397] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 16:38:26.172030 15086 tablet_bootstrap.cc:492] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c: No bootstrap required, opened a new log
I20260928 16:38:26.172202 15086 ts_tablet_manager.cc:1397] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 16:38:26.174775 15085 raft_consensus.cc:348] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dc74c19043467a9fa5a9399bcee4b5" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 39501 } } peers { permanent_uuid: "0a24e0afa6dc4da886897c9447dba506" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39523 } } peers { permanent_uuid: "cc7d7cffd8854020899b724e7c6bbc1c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 40763 } }
I20260928 16:38:26.175014 15085 raft_consensus.cc:374] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:26.175084 15085 raft_consensus.cc:729] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a24e0afa6dc4da886897c9447dba506, State: Initialized, Role: FOLLOWER
I20260928 16:38:26.175194 15084 raft_consensus.cc:348] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dc74c19043467a9fa5a9399bcee4b5" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 39501 } } peers { permanent_uuid: "0a24e0afa6dc4da886897c9447dba506" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39523 } } peers { permanent_uuid: "cc7d7cffd8854020899b724e7c6bbc1c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 40763 } }
I20260928 16:38:26.175441 15084 raft_consensus.cc:374] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:26.175534 15084 raft_consensus.cc:729] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84dc74c19043467a9fa5a9399bcee4b5, State: Initialized, Role: FOLLOWER
I20260928 16:38:26.175547 15085 consensus_queue.cc:260] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dc74c19043467a9fa5a9399bcee4b5" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 39501 } } peers { permanent_uuid: "0a24e0afa6dc4da886897c9447dba506" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39523 } } peers { permanent_uuid: "cc7d7cffd8854020899b724e7c6bbc1c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 40763 } }
I20260928 16:38:26.176203 14796 heartbeater.cc:503] Master 127.27.122.254:40473 was elected leader, sending a full tablet report...
I20260928 16:38:26.176362 15086 raft_consensus.cc:348] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dc74c19043467a9fa5a9399bcee4b5" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 39501 } } peers { permanent_uuid: "0a24e0afa6dc4da886897c9447dba506" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39523 } } peers { permanent_uuid: "cc7d7cffd8854020899b724e7c6bbc1c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 40763 } }
I20260928 16:38:26.176534 15085 ts_tablet_manager.cc:1428] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:38:26.176601 15086 raft_consensus.cc:374] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:26.176693 15086 raft_consensus.cc:729] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc7d7cffd8854020899b724e7c6bbc1c, State: Initialized, Role: FOLLOWER
I20260928 16:38:26.175968 15084 consensus_queue.cc:260] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dc74c19043467a9fa5a9399bcee4b5" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 39501 } } peers { permanent_uuid: "0a24e0afa6dc4da886897c9447dba506" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39523 } } peers { permanent_uuid: "cc7d7cffd8854020899b724e7c6bbc1c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 40763 } }
I20260928 16:38:26.177134 15086 consensus_queue.cc:260] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dc74c19043467a9fa5a9399bcee4b5" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 39501 } } peers { permanent_uuid: "0a24e0afa6dc4da886897c9447dba506" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39523 } } peers { permanent_uuid: "cc7d7cffd8854020899b724e7c6bbc1c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 40763 } }
I20260928 16:38:26.177577 14931 heartbeater.cc:503] Master 127.27.122.254:40473 was elected leader, sending a full tablet report...
I20260928 16:38:26.177856 15084 ts_tablet_manager.cc:1428] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260928 16:38:26.178081 15086 ts_tablet_manager.cc:1428] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 16:38:26.178428 15067 heartbeater.cc:503] Master 127.27.122.254:40473 was elected leader, sending a full tablet report...
W20260928 16:38:26.241494 14797 tablet.cc:2369] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:38:26.334682 15091 raft_consensus.cc:482] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:38:26.334895 15091 raft_consensus.cc:504] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dc74c19043467a9fa5a9399bcee4b5" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 39501 } } peers { permanent_uuid: "0a24e0afa6dc4da886897c9447dba506" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39523 } } peers { permanent_uuid: "cc7d7cffd8854020899b724e7c6bbc1c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 40763 } }
I20260928 16:38:26.336161 15091 leader_election.cc:288] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a24e0afa6dc4da886897c9447dba506 (127.27.122.193:39523), cc7d7cffd8854020899b724e7c6bbc1c (127.27.122.195:40763)
I20260928 16:38:26.340124 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.336508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:39523 (local address 127.27.122.194:54333)
0928 16:38:26.336624 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:26.336641 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:26.336737 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:26.337200 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:26.337203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:26.337223 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:26.337437 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:26.337446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.338216 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.338223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:26.339031 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:26.339040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.339701 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.339704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:26.339804 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.339826 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:38:26.339916 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 16:38:26.339988 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 16:38:26.340124 15083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.336742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:54333 (local address 127.27.122.193:39523)
0928 16:38:26.336872 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:26.336876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:26.336888 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:38:26.336988 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:26.336991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:26.337044 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:26.337153 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:26.337571 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.338077 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.339161 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.339579 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.339707 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.339759 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:26.339934 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 16:38:26.340003 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:38:26.340667 15093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.336516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:40763 (local address 127.27.122.194:55603)
0928 16:38:26.337261 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:26.337272 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:26.337393 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:26.337656 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:26.337659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:26.337674 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:26.337799 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:26.337804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.338634 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.338638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:26.339448 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:26.339457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.340209 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.340213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:26.340315 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.340342 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 16:38:26.340453 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 16:38:26.340529 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":298,"thread_start_us":191,"threads_started":1}
I20260928 16:38:26.340672 15082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.336620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:55603 (local address 127.27.122.195:40763)
0928 16:38:26.336742 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:26.336745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:26.337392 (+ 647us) server_negotiation.cc:407] Connection header received
0928 16:38:26.337498 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:26.337501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:26.337551 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:26.337630 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:26.337918 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.338526 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.339577 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.340075 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.340209 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.340253 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:26.340487 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 16:38:26.340581 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 16:38:26.340772 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9712d90645914cd8b2d4c9fb909f9ff9" candidate_uuid: "84dc74c19043467a9fa5a9399bcee4b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a24e0afa6dc4da886897c9447dba506" is_pre_election: true
I20260928 16:38:26.341112 14749 raft_consensus.cc:2454] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84dc74c19043467a9fa5a9399bcee4b5 in term 0.
I20260928 16:38:26.341548 14814 leader_election.cc:302] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a24e0afa6dc4da886897c9447dba506, 84dc74c19043467a9fa5a9399bcee4b5; no voters:
I20260928 16:38:26.341826 15091 raft_consensus.cc:2781] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:38:26.341914 15091 raft_consensus.cc:482] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:38:26.341864 15019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9712d90645914cd8b2d4c9fb909f9ff9" candidate_uuid: "84dc74c19043467a9fa5a9399bcee4b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7d7cffd8854020899b724e7c6bbc1c" is_pre_election: true
I20260928 16:38:26.341946 15091 raft_consensus.cc:3037] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:26.342139 15019 raft_consensus.cc:2454] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84dc74c19043467a9fa5a9399bcee4b5 in term 0.
I20260928 16:38:26.343006 15091 raft_consensus.cc:504] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dc74c19043467a9fa5a9399bcee4b5" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 39501 } } peers { permanent_uuid: "0a24e0afa6dc4da886897c9447dba506" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39523 } } peers { permanent_uuid: "cc7d7cffd8854020899b724e7c6bbc1c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 40763 } }
I20260928 16:38:26.343362 15091 leader_election.cc:288] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [CANDIDATE]: Term 1 election: Requested vote from peers 0a24e0afa6dc4da886897c9447dba506 (127.27.122.193:39523), cc7d7cffd8854020899b724e7c6bbc1c (127.27.122.195:40763)
I20260928 16:38:26.343701 14749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9712d90645914cd8b2d4c9fb909f9ff9" candidate_uuid: "84dc74c19043467a9fa5a9399bcee4b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a24e0afa6dc4da886897c9447dba506"
I20260928 16:38:26.343824 14749 raft_consensus.cc:3037] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:26.343860 15019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9712d90645914cd8b2d4c9fb909f9ff9" candidate_uuid: "84dc74c19043467a9fa5a9399bcee4b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc7d7cffd8854020899b724e7c6bbc1c"
I20260928 16:38:26.343987 15019 raft_consensus.cc:3037] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:26.344858 14749 raft_consensus.cc:2454] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84dc74c19043467a9fa5a9399bcee4b5 in term 1.
I20260928 16:38:26.345155 15019 raft_consensus.cc:2454] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84dc74c19043467a9fa5a9399bcee4b5 in term 1.
I20260928 16:38:26.345243 14814 leader_election.cc:302] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a24e0afa6dc4da886897c9447dba506, 84dc74c19043467a9fa5a9399bcee4b5; no voters:
I20260928 16:38:26.345554 15091 raft_consensus.cc:2781] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:26.345777 15091 raft_consensus.cc:686] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [term 1 LEADER]: Becoming Leader. State: Replica: 84dc74c19043467a9fa5a9399bcee4b5, State: Running, Role: LEADER
I20260928 16:38:26.346101 15091 consensus_queue.cc:237] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dc74c19043467a9fa5a9399bcee4b5" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 39501 } } peers { permanent_uuid: "0a24e0afa6dc4da886897c9447dba506" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39523 } } peers { permanent_uuid: "cc7d7cffd8854020899b724e7c6bbc1c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 40763 } }
I20260928 16:38:26.349273 14602 catalog_manager.cc:5495] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84dc74c19043467a9fa5a9399bcee4b5 (127.27.122.194). New cstate: current_term: 1 leader_uuid: "84dc74c19043467a9fa5a9399bcee4b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84dc74c19043467a9fa5a9399bcee4b5" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 39501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a24e0afa6dc4da886897c9447dba506" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc7d7cffd8854020899b724e7c6bbc1c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 40763 } health_report { overall_health: UNKNOWN } } }
W20260928 16:38:26.361290 15068 tablet.cc:2369] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:38:26.389315 15079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.385654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.27.122.194:39501)
0928 16:38:26.385808 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:26.385813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:26.385829 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:38:26.385930 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:26.385932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:26.385989 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:26.386105 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:38:26.386366 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.386937 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.387797 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:26.388011 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:26.388117 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:26.388444 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:38:26.388853 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:38:26.389043 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:26.389132 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 16:38:26.389170 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:38:26.395473 15019 raft_consensus.cc:1261] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c [term 1 FOLLOWER]: Refusing update from remote peer 84dc74c19043467a9fa5a9399bcee4b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:26.395473 14749 raft_consensus.cc:1261] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506 [term 1 FOLLOWER]: Refusing update from remote peer 84dc74c19043467a9fa5a9399bcee4b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:26.396265 15094 consensus_queue.cc:1035] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc7d7cffd8854020899b724e7c6bbc1c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 40763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:38:26.396543 15091 consensus_queue.cc:1035] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a24e0afa6dc4da886897c9447dba506" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 39523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:38:26.412156 14601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:38:26.412324 14601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:38:26.414836 14601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ecf3ebb2b3b14aaf875963550da17d87: Sending DeleteTablet for 3 replicas of tablet 9712d90645914cd8b2d4c9fb909f9ff9
I20260928 16:38:26.415606 14999 tablet_service.cc:1514] Processing DeleteTablet for tablet 9712d90645914cd8b2d4c9fb909f9ff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:26 UTC) from {username='slave'} at 127.0.0.1:35660
I20260928 16:38:26.415621 14862 tablet_service.cc:1514] Processing DeleteTablet for tablet 9712d90645914cd8b2d4c9fb909f9ff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:26 UTC) from {username='slave'} at 127.0.0.1:49062
I20260928 16:38:26.415655 14729 tablet_service.cc:1514] Processing DeleteTablet for tablet 9712d90645914cd8b2d4c9fb909f9ff9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:26 UTC) from {username='slave'} at 127.0.0.1:60232
I20260928 16:38:26.416069 15109 tablet_replica.cc:330] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5: stopping tablet replica
I20260928 16:38:26.416204 15108 tablet_replica.cc:330] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c: stopping tablet replica
I20260928 16:38:26.416425 15109 raft_consensus.cc:2229] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:38:26.416704 15107 tablet_replica.cc:330] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506: stopping tablet replica
I20260928 16:38:26.416751 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 14664
I20260928 16:38:26.417023 15107 raft_consensus.cc:2229] T 9712d90645914cd8b2d4c9fb909f9ff9 P 0a24e0afa6dc4da886897c9447dba506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:38:26.418553 15109 raft_consensus.cc:2258] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:26.419600 15108 raft_consensus.cc:2229] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:38:26.419909 15108 raft_consensus.cc:2258] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:26.422004 15108 ts_tablet_manager.cc:1905] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:38:26.422564 15109 ts_tablet_manager.cc:1905] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:38:26.424757 15108 ts_tablet_manager.cc:1918] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:38:26.424885 15108 log.cc:1196] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/9712d90645914cd8b2d4c9fb909f9ff9
I20260928 16:38:26.425239 15108 ts_tablet_manager.cc:1939] T 9712d90645914cd8b2d4c9fb909f9ff9 P cc7d7cffd8854020899b724e7c6bbc1c: Deleting consensus metadata
W20260928 16:38:26.425449 14586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:39523 (error 108)
I20260928 16:38:26.425989 15109 ts_tablet_manager.cc:1918] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:38:26.426146 15109 log.cc:1196] T 9712d90645914cd8b2d4c9fb909f9ff9 P 84dc74c19043467a9fa5a9399bcee4b5: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/9712d90645914cd8b2d4c9fb909f9ff9
I20260928 16:38:26.426267 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 14799
I20260928 16:38:26.426437 14588 catalog_manager.cc:4841] TS cc7d7cffd8854020899b724e7c6bbc1c (127.27.122.195:40763): tablet 9712d90645914cd8b2d4c9fb909f9ff9 (table test-table [id=84dda325c0fa4b088b5a4b3f880f6cbb]) successfully deleted
I20260928 16:38:26.428197 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.427901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:39523 (local address 127.0.0.1:60236)
0928 16:38:26.428031 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:26.428112 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:39523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 16:38:26.428422 14586 catalog_manager.cc:4584] TS 0a24e0afa6dc4da886897c9447dba506 (127.27.122.193:39523): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9712d90645914cd8b2d4c9fb909f9ff9: Network error: Client connection negotiation failed: client connection to 127.27.122.193:39523: connect: Connection refused (error 111)
I20260928 16:38:26.433141 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 14934
I20260928 16:38:26.433946 15078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:26.433297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:39501 (local address 127.0.0.1:49082)
0928 16:38:26.433792 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:26.433852 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.194:39501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403}
I20260928 16:38:26.439246 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 14571
2026-09-28T16:38:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1120 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:26 dist-test-slave-g9sh krb5kdc[15113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:26 dist-test-slave-g9sh krb5kdc[15113](info): set up 2 sockets
Sep 28 16:38:26 dist-test-slave-g9sh krb5kdc[15113](info): commencing operation
krb5kdc: starting...
W20260928 16:38:27.467693 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:27 dist-test-slave-g9sh krb5kdc[15113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:38:27.586265 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:38573
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:38573
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:27.694397 15126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:27.694687 15126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:27.694774 15126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:27.698757 15126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:27.698846 15126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:27.698887 15126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:27.698947 15126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:27.699005 15126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:27.704222 15126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:38573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:38573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:27.705245 15126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:27.706247 15126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:27.714701 15131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:27.714756 15134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:27.714767 15132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:27.714864 15126 server_base.cc:1034] running on GCE node
I20260928 16:38:27.715878 15126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:27.716961 15126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:27.718149 15126 hybrid_clock.cc:648] HybridClock initialized: now 1790613507718142 us; error 29 us; skew 500 ppm
Sep 28 16:38:27 dist-test-slave-g9sh krb5kdc[15113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613507, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:27.720794 15126 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:38:27.721724 15126 webserver.cc:460] Webserver started at http://127.27.122.254:40231/ using document root <none> and password file <none>
I20260928 16:38:27.722270 15126 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:27.722371 15126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:27.722620 15126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:27.724610 15126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "0315993ac8d7479f953de60e5e622bf6"
format_stamp: "Formatted at 2026-09-28 16:38:27 on dist-test-slave-g9sh"
server_key: "fcb8ba8be54a4d1fedd96993e82f3b42"
server_key_iv: "892653db7307bfd01ccf345322f96f87"
server_key_version: "encryptionkey@0"
I20260928 16:38:27.725216 15126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "0315993ac8d7479f953de60e5e622bf6"
format_stamp: "Formatted at 2026-09-28 16:38:27 on dist-test-slave-g9sh"
server_key: "fcb8ba8be54a4d1fedd96993e82f3b42"
server_key_iv: "892653db7307bfd01ccf345322f96f87"
server_key_version: "encryptionkey@0"
I20260928 16:38:27.728775 15126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:38:27.731156 15141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:27.732426 15126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:38:27.732617 15126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "0315993ac8d7479f953de60e5e622bf6"
format_stamp: "Formatted at 2026-09-28 16:38:27 on dist-test-slave-g9sh"
server_key: "fcb8ba8be54a4d1fedd96993e82f3b42"
server_key_iv: "892653db7307bfd01ccf345322f96f87"
server_key_version: "encryptionkey@0"
I20260928 16:38:27.732738 15126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:27.747694 15126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:27.750757 15126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:27.750957 15126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:27.758021 15126 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:38573
I20260928 16:38:27.758041 15194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:38573 every 8 connection(s)
I20260928 16:38:27.758930 15126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:27.761679 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 15126
I20260928 16:38:27.761699 15195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:27.761813 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:27.762094 28139 external_mini_cluster.cc:1428] Setting key d69290a1cf606735c7f343b9c2051168
I20260928 16:38:27.767797 15195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6: Bootstrap starting.
I20260928 16:38:27.769941 15195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:27.770784 15195 log.cc:824] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:38:27 dist-test-slave-g9sh krb5kdc[15113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:27.773121 15195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6: No bootstrap required, opened a new log
I20260928 16:38:27.776553 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:27.763452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48550 (local address 127.27.122.254:38573)
0928 16:38:27.763896 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:27.763909 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:27.763944 (+ 35us) server_negotiation.cc:407] Connection header received
0928 16:38:27.764823 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:27.764841 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:27.765226 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:27.765589 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:27.766376 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:27.767335 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:27.768258 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:27.768722 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:27.768885 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:27.771682 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:27.771691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:27.771709 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:27.771736 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:27.773607 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:27.774010 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:27.774016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:27.774022 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:27.774155 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:27.774402 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:27.774406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:27.774408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:27.774887 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:27.775087 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:27.775302 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 16:38:27.775651 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":140,"threads_started":1}
I20260928 16:38:27.777601 15195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0315993ac8d7479f953de60e5e622bf6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38573 } }
I20260928 16:38:27.777815 15195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:27.777879 15195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315993ac8d7479f953de60e5e622bf6, State: Initialized, Role: FOLLOWER
I20260928 16:38:27.778326 15195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0315993ac8d7479f953de60e5e622bf6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38573 } }
I20260928 16:38:27.778441 15195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:27.778520 15195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:27.778630 15195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:27.779538 15195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0315993ac8d7479f953de60e5e622bf6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38573 } }
I20260928 16:38:27.779811 15195 leader_election.cc:302] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0315993ac8d7479f953de60e5e622bf6; no voters:
I20260928 16:38:27.780046 15195 leader_election.cc:288] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:27.780174 15200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:27.780416 15200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 0315993ac8d7479f953de60e5e622bf6, State: Running, Role: LEADER
I20260928 16:38:27.780769 15200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0315993ac8d7479f953de60e5e622bf6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38573 } }
I20260928 16:38:27.781239 15195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:27.782356 15201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0315993ac8d7479f953de60e5e622bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0315993ac8d7479f953de60e5e622bf6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38573 } } }
I20260928 16:38:27.782482 15201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:27.782676 15202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0315993ac8d7479f953de60e5e622bf6. Latest consensus state: current_term: 1 leader_uuid: "0315993ac8d7479f953de60e5e622bf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0315993ac8d7479f953de60e5e622bf6" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38573 } } }
I20260928 16:38:27.782863 15202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:27.783602 15210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:27.786638 15210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:27.791427 15210 catalog_manager.cc:1347] Generated new cluster ID: 11ef7188b0c54dd09aa87d6716c09328
I20260928 16:38:27.791530 15210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:27.800352 15210 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:27.801247 15210 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:27.808651 15210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6: Generated new TSK 0
I20260928 16:38:27.809288 15210 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.122.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
Entry for principal kudu/127.27.122.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab.
I20260928 16:38:27.844290 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:38573
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:27.953423 15221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:27.953644 15221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:27.953725 15221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:27.957944 15221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:27.957999 15221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:27.958055 15221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:38:27.963762 15221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.193.keytab
--principal=kudu/127.27.122.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:38573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:27.964624 15221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:27.965552 15221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:38:27.974843 15221 server_base.cc:1034] running on GCE node
W20260928 16:38:27.974881 15227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:27.974840 15229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:27.975006 15226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:27.975437 15221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:27.976037 15221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:27.977217 15221 hybrid_clock.cc:648] HybridClock initialized: now 1790613507977197 us; error 31 us; skew 500 ppm
Sep 28 16:38:27 dist-test-slave-g9sh krb5kdc[15113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613507, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:27.979825 15221 init.cc:377] Logged in from keytab as kudu/127.27.122.193@KRBTEST.COM (short username kudu)
I20260928 16:38:27.980600 15221 webserver.cc:460] Webserver started at http://127.27.122.193:42249/ using document root <none> and password file <none>
I20260928 16:38:27.981197 15221 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:27.981284 15221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:27.981561 15221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:27.983669 15221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "7903a45d29fa4156b3644783881840c0"
format_stamp: "Formatted at 2026-09-28 16:38:27 on dist-test-slave-g9sh"
server_key: "4b436d9f7854cacec695ce74f833a1bd"
server_key_iv: "a95923c61f4c6168b7de89e5f9ec02af"
server_key_version: "encryptionkey@0"
I20260928 16:38:27.984200 15221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "7903a45d29fa4156b3644783881840c0"
format_stamp: "Formatted at 2026-09-28 16:38:27 on dist-test-slave-g9sh"
server_key: "4b436d9f7854cacec695ce74f833a1bd"
server_key_iv: "a95923c61f4c6168b7de89e5f9ec02af"
server_key_version: "encryptionkey@0"
I20260928 16:38:27.987731 15221 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:38:27.990078 15236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:27.991215 15221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:38:27.991346 15221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "7903a45d29fa4156b3644783881840c0"
format_stamp: "Formatted at 2026-09-28 16:38:27 on dist-test-slave-g9sh"
server_key: "4b436d9f7854cacec695ce74f833a1bd"
server_key_iv: "a95923c61f4c6168b7de89e5f9ec02af"
server_key_version: "encryptionkey@0"
I20260928 16:38:27.991449 15221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:28.011866 15221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:28.014730 15221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:28.014930 15221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:28.015525 15221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:28.016448 15221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:28.016539 15221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:28.016634 15221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:28.016688 15221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:28.025518 15221 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:35951
I20260928 16:38:28.025561 15352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:35951 every 8 connection(s)
I20260928 16:38:28.026322 15221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:38:28.030162 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 15221
I20260928 16:38:28.030303 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:38:28.030618 28139 external_mini_cluster.cc:1428] Setting key 616947b5527ee0e4ecbfe45ed2198b97
Sep 28 16:38:28 dist-test-slave-g9sh krb5kdc[15113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613507, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.193@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.194@KRBTEST.COM; defaulting to no policy
I20260928 16:38:28.041158 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.028053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:36885 (local address 127.27.122.254:38573)
0928 16:38:28.028210 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:28.028216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:28.029058 (+ 842us) server_negotiation.cc:407] Connection header received
0928 16:38:28.030346 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:28.030349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:28.030410 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:28.030518 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:28.031797 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.032438 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.033796 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.034030 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.034118 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.037164 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:28.037171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:28.037176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:28.037213 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:28.038957 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:28.039485 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:28.039491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:28.039493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:28.039591 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:28.039924 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:28.039927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:28.039929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:28.040198 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:28.040309 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:28.040776 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 16:38:28.040943 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:38:28.041857 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.028395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38573 (local address 127.27.122.193:36885)
0928 16:38:28.028852 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:28.028893 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:28.030129 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:28.030713 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:28.030724 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:28.031170 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:28.031653 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.031666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.032574 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.032578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.033666 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.033681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.034187 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.034193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.034410 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.035080 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:28.035095 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:28.036987 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:28.039075 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:28.039080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:28.039092 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:28.039365 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:28.039687 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:28.039690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:28.039691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:28.039817 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:28.040305 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:28.040310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:28.040527 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 16:38:28.040676 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 16:38:28.040977 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":120,"threads_started":1}
I20260928 16:38:28.043205 15353 heartbeater.cc:347] Connected to a master server at 127.27.122.254:38573
I20260928 16:38:28.043444 15353 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.194@KRBTEST.COM" created.
I20260928 16:38:28.044019 15353 heartbeater.cc:511] Master 127.27.122.254:38573 requested a full tablet report, sending...
I20260928 16:38:28.045800 15158 ts_manager.cc:194] Registered new tserver with Master: 7903a45d29fa4156b3644783881840c0 (127.27.122.193:35951)
I20260928 16:38:28.047030 15158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.193@KRBTEST.COM'} at 127.27.122.193:36885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
Entry for principal kudu/127.27.122.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab.
I20260928 16:38:28.059808 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:38573
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:28.168468 15358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:28.168711 15358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:28.168752 15358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:28.172855 15358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:28.172956 15358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:28.173055 15358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:38:28.178231 15358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.194.keytab
--principal=kudu/127.27.122.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:38573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:28.179214 15358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:28.180197 15358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:28.189841 15366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:28.189853 15363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:28.189874 15364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:28.189917 15358 server_base.cc:1034] running on GCE node
I20260928 16:38:28.190485 15358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:28.191068 15358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:28.192270 15358 hybrid_clock.cc:648] HybridClock initialized: now 1790613508192249 us; error 34 us; skew 500 ppm
Sep 28 16:38:28 dist-test-slave-g9sh krb5kdc[15113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613508, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:28.195012 15358 init.cc:377] Logged in from keytab as kudu/127.27.122.194@KRBTEST.COM (short username kudu)
I20260928 16:38:28.195931 15358 webserver.cc:460] Webserver started at http://127.27.122.194:45321/ using document root <none> and password file <none>
I20260928 16:38:28.196481 15358 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:28.196547 15358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:28.196774 15358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:28.198792 15358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "d77920ddcbc1490d9762ee1165e5a54e"
format_stamp: "Formatted at 2026-09-28 16:38:28 on dist-test-slave-g9sh"
server_key: "e36be02ae8306a7a39e2591d5196d89f"
server_key_iv: "9a22dc4945c18fd869bef8ae106be1e6"
server_key_version: "encryptionkey@0"
I20260928 16:38:28.199375 15358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "d77920ddcbc1490d9762ee1165e5a54e"
format_stamp: "Formatted at 2026-09-28 16:38:28 on dist-test-slave-g9sh"
server_key: "e36be02ae8306a7a39e2591d5196d89f"
server_key_iv: "9a22dc4945c18fd869bef8ae106be1e6"
server_key_version: "encryptionkey@0"
I20260928 16:38:28.202942 15358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:38:28.205279 15373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:28.206472 15358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:28.206588 15358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "d77920ddcbc1490d9762ee1165e5a54e"
format_stamp: "Formatted at 2026-09-28 16:38:28 on dist-test-slave-g9sh"
server_key: "e36be02ae8306a7a39e2591d5196d89f"
server_key_iv: "9a22dc4945c18fd869bef8ae106be1e6"
server_key_version: "encryptionkey@0"
I20260928 16:38:28.206687 15358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:28.226670 15358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:28.229673 15358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:28.229887 15358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:28.230492 15358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:28.231317 15358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:28.231365 15358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:28.231444 15358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:28.231482 15358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:28.240789 15358 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:44753
I20260928 16:38:28.240814 15488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:44753 every 8 connection(s)
I20260928 16:38:28.241696 15358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:38:28.245428 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 15358
I20260928 16:38:28.245615 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:38:28.245937 28139 external_mini_cluster.cc:1428] Setting key c941ca00c21a405013c873377bbcf2b5
Sep 28 16:38:28 dist-test-slave-g9sh krb5kdc[15113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613508, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.194@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.122.195@KRBTEST.COM; defaulting to no policy
I20260928 16:38:28.256644 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.243471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:49285 (local address 127.27.122.254:38573)
0928 16:38:28.243613 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:28.243616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:28.244480 (+ 864us) server_negotiation.cc:407] Connection header received
0928 16:38:28.245945 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:28.245952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:28.246017 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:28.246117 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:28.247449 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.248213 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.249492 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.249889 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.250019 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.252795 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:28.252799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:28.252801 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:28.252826 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:28.254354 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:28.254964 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:28.254967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:28.254968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:28.255050 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:28.255473 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:28.255476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:28.255478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:28.255663 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:28.255743 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:28.256399 (+ 656us) server_negotiation.cc:299] Negotiation successful
0928 16:38:28.256520 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 16:38:28.257591 15491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.243860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38573 (local address 127.27.122.194:49285)
0928 16:38:28.244325 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:28.244369 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:28.245713 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:28.246288 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:28.246295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:28.246750 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:28.247272 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.247294 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.248351 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.248354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.249367 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.249378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.250045 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.250047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.250194 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.250787 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:28.250804 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:28.252615 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:28.254481 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:28.254487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:28.254503 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:28.254861 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:28.255150 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:28.255154 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:28.255156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:28.255359 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:28.255782 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:28.255791 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:28.256143 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 16:38:28.256310 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 16:38:28.256559 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.195@KRBTEST.COM" created.
I20260928 16:38:28.258888 15489 heartbeater.cc:347] Connected to a master server at 127.27.122.254:38573
I20260928 16:38:28.259128 15489 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:28.259682 15489 heartbeater.cc:511] Master 127.27.122.254:38573 requested a full tablet report, sending...
I20260928 16:38:28.260886 15158 ts_manager.cc:194] Registered new tserver with Master: d77920ddcbc1490d9762ee1165e5a54e (127.27.122.194:44753)
I20260928 16:38:28.261555 15158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.194@KRBTEST.COM'} at 127.27.122.194:49285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
Entry for principal kudu/127.27.122.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab.
I20260928 16:38:28.274019 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:38573
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:40895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:28.382789 15494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:28.383066 15494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:28.383141 15494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:28.387228 15494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:28.387318 15494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:28.387415 15494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:38:28.392540 15494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:40895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.195.keytab
--principal=kudu/127.27.122.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:38573
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:28.393606 15494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:28.394671 15494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:28.404372 15502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:28.404421 15494 server_base.cc:1034] running on GCE node
W20260928 16:38:28.404421 15500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:28.404381 15499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:28.405141 15494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:28.405763 15494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:28.406955 15494 hybrid_clock.cc:648] HybridClock initialized: now 1790613508406929 us; error 35 us; skew 500 ppm
Sep 28 16:38:28 dist-test-slave-g9sh krb5kdc[15113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613508, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:28.409716 15494 init.cc:377] Logged in from keytab as kudu/127.27.122.195@KRBTEST.COM (short username kudu)
I20260928 16:38:28.410616 15494 webserver.cc:460] Webserver started at http://127.27.122.195:36365/ using document root <none> and password file <none>
I20260928 16:38:28.411128 15494 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:28.411188 15494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:28.411350 15494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:28.413357 15494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "33d07e286b1e402c8d93bab96a9f7f4c"
format_stamp: "Formatted at 2026-09-28 16:38:28 on dist-test-slave-g9sh"
server_key: "bcdae97be0733184d22c063fbd855960"
server_key_iv: "932792b4d73354917b18747e1066013c"
server_key_version: "encryptionkey@0"
I20260928 16:38:28.413869 15494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "33d07e286b1e402c8d93bab96a9f7f4c"
format_stamp: "Formatted at 2026-09-28 16:38:28 on dist-test-slave-g9sh"
server_key: "bcdae97be0733184d22c063fbd855960"
server_key_iv: "932792b4d73354917b18747e1066013c"
server_key_version: "encryptionkey@0"
I20260928 16:38:28.417446 15494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:38:28.419883 15509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:28.421244 15494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:38:28.421376 15494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "33d07e286b1e402c8d93bab96a9f7f4c"
format_stamp: "Formatted at 2026-09-28 16:38:28 on dist-test-slave-g9sh"
server_key: "bcdae97be0733184d22c063fbd855960"
server_key_iv: "932792b4d73354917b18747e1066013c"
server_key_version: "encryptionkey@0"
I20260928 16:38:28.421489 15494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:28.445375 15494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:28.448348 15494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:28.448560 15494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:28.449182 15494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:28.450065 15494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:28.450137 15494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:28.450213 15494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:28.450256 15494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:28.459410 15494 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:42275
I20260928 16:38:28.459434 15624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:42275 every 8 connection(s)
I20260928 16:38:28.460341 15494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
Sep 28 16:38:28 dist-test-slave-g9sh krb5kdc[15113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613508, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.195@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:28.469859 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 15494
I20260928 16:38:28.470000 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:38:28.470327 28139 external_mini_cluster.cc:1428] Setting key 96f0c351ca591baef8062c1597af734a
I20260928 16:38:28.474074 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.462154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:41135 (local address 127.27.122.254:38573)
0928 16:38:28.462292 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:28.462296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:28.463171 (+ 875us) server_negotiation.cc:407] Connection header received
0928 16:38:28.464479 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:28.464483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:28.464543 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:28.464663 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:28.465860 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.466399 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.467543 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.467778 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.467889 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.470493 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:28.470496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:28.470498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:28.470524 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:28.472114 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:28.472724 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:28.472727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:28.472728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:28.472805 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:28.473162 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:28.473165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:28.473166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:28.473329 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:28.473404 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:28.473863 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 16:38:28.473956 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:38:28.474896 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.462580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:38573 (local address 127.27.122.195:41135)
0928 16:38:28.463019 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:28.463069 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:28.464259 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:28.464841 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:28.464848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:28.465242 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:28.465716 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.465729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.466529 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.466537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.467415 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.467424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.467931 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.467935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.468109 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.468678 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:28.468694 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:28.470302 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:28.472237 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:28.472242 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:28.472257 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:28.472631 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:28.472886 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:28.472888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:28.472889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:28.473081 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:28.473417 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:28.473422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:28.473659 (+ 237us) client_negotiation.cc:769] Sending connection context
0928 16:38:28.473801 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 16:38:28.474044 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260928 16:38:28.475917 15625 heartbeater.cc:347] Connected to a master server at 127.27.122.254:38573
I20260928 16:38:28.476158 15625 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:28.476645 15625 heartbeater.cc:511] Master 127.27.122.254:38573 requested a full tablet report, sending...
I20260928 16:38:28.477662 15158 ts_manager.cc:194] Registered new tserver with Master: 33d07e286b1e402c8d93bab96a9f7f4c (127.27.122.195:42275)
I20260928 16:38:28.478222 15158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.122.195@KRBTEST.COM'} at 127.27.122.195:41135
I20260928 16:38:28.484282 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:38:28.493399 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.486232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.27.122.254:38573)
0928 16:38:28.486368 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:28.486374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:28.486579 (+ 205us) server_negotiation.cc:407] Connection header received
0928 16:38:28.486678 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:28.486680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:28.486718 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:28.486781 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:28.487401 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.487899 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.488685 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.488885 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.489011 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.490211 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:28.490213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:28.490215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:28.490237 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:28.491643 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:28.492187 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:28.492192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:28.492193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:28.492302 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:28.492593 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:28.492595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:28.492596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:28.492797 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:28.492941 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:28.493127 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 16:38:28.493215 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:38:28.496172 15158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:38:28.499471 15158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:38:28.513821 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.507743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:42275 (local address 127.0.0.1:36800)
0928 16:38:28.508009 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:28.508024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:28.508137 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:28.509195 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:28.509202 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:28.509246 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:28.509477 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.509485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.510861 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.510864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.512339 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.512349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.513238 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.513241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.513367 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.513442 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 16:38:28.513547 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 16:38:28.513664 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":49,"spinlock_wait_cycles":7168,"thread_start_us":114,"threads_started":1}
I20260928 16:38:28.513971 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.507537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38126 (local address 127.27.122.193:35951)
0928 16:38:28.508743 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:28.508756 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:28.508775 (+ 19us) server_negotiation.cc:407] Connection header received
0928 16:38:28.508839 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:28.508843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:28.509012 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:28.509324 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:28.510296 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.511213 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.512535 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.513429 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.513549 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.513681 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:28.513744 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 16:38:28.513828 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1107,"thread_start_us":112,"threads_started":1}
I20260928 16:38:28.513883 15636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.507359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:35951 (local address 127.0.0.1:38126)
0928 16:38:28.507759 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:28.507812 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:28.507934 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:28.509195 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:28.509202 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:28.509245 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:28.509492 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.509498 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.511340 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.511346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.512395 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.512404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.513531 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.513537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.513631 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.513652 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:38:28.513706 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 16:38:28.513765 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":86,"spinlock_wait_cycles":5248,"thread_start_us":130,"threads_started":1}
I20260928 16:38:28.514575 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.508327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.27.122.195:42275)
0928 16:38:28.508658 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:28.508671 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:28.508695 (+ 24us) server_negotiation.cc:407] Connection header received
0928 16:38:28.508770 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:28.508775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:28.509012 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:28.509353 (+ 341us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:28.509609 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.510730 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.512462 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.513109 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.513239 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.513402 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:28.514294 (+ 892us) server_negotiation.cc:299] Negotiation successful
0928 16:38:28.514400 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":136,"threads_started":1}
I20260928 16:38:28.515303 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.508392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:44753 (local address 127.0.0.1:58362)
0928 16:38:28.509101 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:28.509114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:28.509205 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:28.510110 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:28.510112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:28.510128 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:28.510225 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.510231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.512130 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.512133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.513265 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.513276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.514910 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.514914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.514997 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.515018 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 16:38:28.515110 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 16:38:28.515186 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":111,"threads_started":1}
I20260928 16:38:28.515383 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.508818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.27.122.194:44753)
0928 16:38:28.509682 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:28.509694 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:28.509715 (+ 21us) server_negotiation.cc:407] Connection header received
0928 16:38:28.509787 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:28.509795 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:28.509978 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:28.510127 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:28.510365 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.511989 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.513439 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.514631 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.514742 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.514851 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:28.515156 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 16:38:28.515252 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":154,"threads_started":1}
I20260928 16:38:28.517096 15558 tablet_service.cc:1467] Processing CreateTablet for tablet 33527dc65c624b818d9623ada1d97623 (DEFAULT_TABLE table=test-table [id=0fe008c5919841f1bf279be4066f3c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:28.517058 15285 tablet_service.cc:1467] Processing CreateTablet for tablet 33527dc65c624b818d9623ada1d97623 (DEFAULT_TABLE table=test-table [id=0fe008c5919841f1bf279be4066f3c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:28.518066 15285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33527dc65c624b818d9623ada1d97623. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:28.518090 15558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33527dc65c624b818d9623ada1d97623. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:28.519594 15421 tablet_service.cc:1467] Processing CreateTablet for tablet 33527dc65c624b818d9623ada1d97623 (DEFAULT_TABLE table=test-table [id=0fe008c5919841f1bf279be4066f3c9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:28.520466 15421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33527dc65c624b818d9623ada1d97623. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:28.524008 15642 tablet_bootstrap.cc:492] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0: Bootstrap starting.
I20260928 16:38:28.525671 15643 tablet_bootstrap.cc:492] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c: Bootstrap starting.
I20260928 16:38:28.526480 15642 tablet_bootstrap.cc:654] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:28.527273 15644 tablet_bootstrap.cc:492] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e: Bootstrap starting.
I20260928 16:38:28.527875 15642 log.cc:824] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:28.527916 15643 tablet_bootstrap.cc:654] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:28.529201 15643 log.cc:824] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:28.529414 15644 tablet_bootstrap.cc:654] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:28.530192 15642 tablet_bootstrap.cc:492] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0: No bootstrap required, opened a new log
I20260928 16:38:28.530382 15642 ts_tablet_manager.cc:1397] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260928 16:38:28.530597 15644 log.cc:824] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:28.530990 15643 tablet_bootstrap.cc:492] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c: No bootstrap required, opened a new log
I20260928 16:38:28.531181 15643 ts_tablet_manager.cc:1397] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:38:28.532459 15644 tablet_bootstrap.cc:492] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e: No bootstrap required, opened a new log
I20260928 16:38:28.532634 15644 ts_tablet_manager.cc:1397] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 16:38:28.533628 15642 raft_consensus.cc:348] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7903a45d29fa4156b3644783881840c0" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35951 } } peers { permanent_uuid: "d77920ddcbc1490d9762ee1165e5a54e" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44753 } } peers { permanent_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42275 } }
I20260928 16:38:28.533857 15642 raft_consensus.cc:374] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:28.533929 15642 raft_consensus.cc:729] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7903a45d29fa4156b3644783881840c0, State: Initialized, Role: FOLLOWER
I20260928 16:38:28.534377 15642 consensus_queue.cc:260] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7903a45d29fa4156b3644783881840c0" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35951 } } peers { permanent_uuid: "d77920ddcbc1490d9762ee1165e5a54e" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44753 } } peers { permanent_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42275 } }
I20260928 16:38:28.534988 15353 heartbeater.cc:503] Master 127.27.122.254:38573 was elected leader, sending a full tablet report...
I20260928 16:38:28.534971 15643 raft_consensus.cc:348] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7903a45d29fa4156b3644783881840c0" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35951 } } peers { permanent_uuid: "d77920ddcbc1490d9762ee1165e5a54e" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44753 } } peers { permanent_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42275 } }
I20260928 16:38:28.535203 15643 raft_consensus.cc:374] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:28.535254 15643 raft_consensus.cc:729] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d07e286b1e402c8d93bab96a9f7f4c, State: Initialized, Role: FOLLOWER
I20260928 16:38:28.535342 15642 ts_tablet_manager.cc:1428] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260928 16:38:28.535667 15643 consensus_queue.cc:260] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7903a45d29fa4156b3644783881840c0" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35951 } } peers { permanent_uuid: "d77920ddcbc1490d9762ee1165e5a54e" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44753 } } peers { permanent_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42275 } }
I20260928 16:38:28.536401 15625 heartbeater.cc:503] Master 127.27.122.254:38573 was elected leader, sending a full tablet report...
I20260928 16:38:28.537068 15644 raft_consensus.cc:348] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7903a45d29fa4156b3644783881840c0" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35951 } } peers { permanent_uuid: "d77920ddcbc1490d9762ee1165e5a54e" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44753 } } peers { permanent_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42275 } }
I20260928 16:38:28.537350 15644 raft_consensus.cc:374] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:28.537416 15644 raft_consensus.cc:729] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77920ddcbc1490d9762ee1165e5a54e, State: Initialized, Role: FOLLOWER
I20260928 16:38:28.537796 15643 ts_tablet_manager.cc:1428] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 16:38:28.537810 15644 consensus_queue.cc:260] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7903a45d29fa4156b3644783881840c0" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35951 } } peers { permanent_uuid: "d77920ddcbc1490d9762ee1165e5a54e" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44753 } } peers { permanent_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42275 } }
I20260928 16:38:28.538641 15644 ts_tablet_manager.cc:1428] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 16:38:28.538785 15489 heartbeater.cc:503] Master 127.27.122.254:38573 was elected leader, sending a full tablet report...
I20260928 16:38:28.643442 15650 raft_consensus.cc:482] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:38:28.643733 15650 raft_consensus.cc:504] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7903a45d29fa4156b3644783881840c0" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35951 } } peers { permanent_uuid: "d77920ddcbc1490d9762ee1165e5a54e" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44753 } } peers { permanent_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42275 } }
I20260928 16:38:28.644954 15650 leader_election.cc:288] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7903a45d29fa4156b3644783881840c0 (127.27.122.193:35951), d77920ddcbc1490d9762ee1165e5a54e (127.27.122.194:44753)
I20260928 16:38:28.649667 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.645338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:35913 (local address 127.27.122.193:35951)
0928 16:38:28.645498 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:28.645502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:28.645520 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:38:28.645663 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:28.645665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:28.645734 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:28.645808 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:28.646133 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.646927 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.648425 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.649073 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.649213 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.649270 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:28.649470 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 16:38:28.649547 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:38:28.649667 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.645669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:56685 (local address 127.27.122.194:44753)
0928 16:38:28.645827 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:28.645832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:28.646002 (+ 170us) server_negotiation.cc:407] Connection header received
0928 16:38:28.646121 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:28.646124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:28.646176 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:28.646255 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:28.646490 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.647177 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.648549 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.649171 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.649269 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.649319 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:28.649474 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 16:38:28.649567 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 16:38:28.649668 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.645206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:35951 (local address 127.27.122.195:35913)
0928 16:38:28.645359 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:28.645379 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:28.645512 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:28.645845 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:28.645848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:28.645874 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:28.646016 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.646022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.647033 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.647036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.648282 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.648292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.649190 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.649192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.649312 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.649342 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 16:38:28.649419 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 16:38:28.649502 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":5}
I20260928 16:38:28.649676 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.645518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:44753 (local address 127.27.122.195:56685)
0928 16:38:28.645906 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:28.645919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:28.646016 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:28.646274 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:28.646276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:28.646291 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:28.646385 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.646389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.647279 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.647282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.648427 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:28.648434 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.649272 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.649277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:28.649355 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.649384 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 16:38:28.649447 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 16:38:28.649502 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":156,"threads_started":1}
I20260928 16:38:28.650331 15441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33527dc65c624b818d9623ada1d97623" candidate_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d77920ddcbc1490d9762ee1165e5a54e" is_pre_election: true
I20260928 16:38:28.650326 15305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33527dc65c624b818d9623ada1d97623" candidate_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7903a45d29fa4156b3644783881840c0" is_pre_election: true
I20260928 16:38:28.650604 15305 raft_consensus.cc:2454] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d07e286b1e402c8d93bab96a9f7f4c in term 0.
I20260928 16:38:28.650619 15441 raft_consensus.cc:2454] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33d07e286b1e402c8d93bab96a9f7f4c in term 0.
I20260928 16:38:28.651129 15511 leader_election.cc:302] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d07e286b1e402c8d93bab96a9f7f4c, 7903a45d29fa4156b3644783881840c0; no voters:
I20260928 16:38:28.651396 15650 raft_consensus.cc:2781] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:38:28.651474 15650 raft_consensus.cc:482] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:38:28.651504 15650 raft_consensus.cc:3037] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:28.652590 15650 raft_consensus.cc:504] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7903a45d29fa4156b3644783881840c0" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35951 } } peers { permanent_uuid: "d77920ddcbc1490d9762ee1165e5a54e" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44753 } } peers { permanent_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42275 } }
I20260928 16:38:28.652974 15650 leader_election.cc:288] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [CANDIDATE]: Term 1 election: Requested vote from peers 7903a45d29fa4156b3644783881840c0 (127.27.122.193:35951), d77920ddcbc1490d9762ee1165e5a54e (127.27.122.194:44753)
I20260928 16:38:28.653256 15305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33527dc65c624b818d9623ada1d97623" candidate_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7903a45d29fa4156b3644783881840c0"
I20260928 16:38:28.653311 15441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33527dc65c624b818d9623ada1d97623" candidate_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d77920ddcbc1490d9762ee1165e5a54e"
I20260928 16:38:28.653362 15305 raft_consensus.cc:3037] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:28.653442 15441 raft_consensus.cc:3037] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:28.654639 15305 raft_consensus.cc:2454] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d07e286b1e402c8d93bab96a9f7f4c in term 1.
I20260928 16:38:28.654727 15441 raft_consensus.cc:2454] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33d07e286b1e402c8d93bab96a9f7f4c in term 1.
I20260928 16:38:28.655052 15511 leader_election.cc:302] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33d07e286b1e402c8d93bab96a9f7f4c, d77920ddcbc1490d9762ee1165e5a54e; no voters:
I20260928 16:38:28.655210 15650 raft_consensus.cc:2781] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:28.655450 15650 raft_consensus.cc:686] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [term 1 LEADER]: Becoming Leader. State: Replica: 33d07e286b1e402c8d93bab96a9f7f4c, State: Running, Role: LEADER
I20260928 16:38:28.655772 15650 consensus_queue.cc:237] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7903a45d29fa4156b3644783881840c0" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35951 } } peers { permanent_uuid: "d77920ddcbc1490d9762ee1165e5a54e" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44753 } } peers { permanent_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42275 } }
I20260928 16:38:28.659000 15157 catalog_manager.cc:5495] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 33d07e286b1e402c8d93bab96a9f7f4c (127.27.122.195). New cstate: current_term: 1 leader_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7903a45d29fa4156b3644783881840c0" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d77920ddcbc1490d9762ee1165e5a54e" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33d07e286b1e402c8d93bab96a9f7f4c" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 42275 } health_report { overall_health: HEALTHY } } }
I20260928 16:38:28.699844 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.696289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.27.122.195:42275)
0928 16:38:28.696443 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:28.696447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:28.696463 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:38:28.696552 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:28.696555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:28.696619 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:28.696725 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:38:28.697015 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.697498 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.698360 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.698587 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.698698 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.699002 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:38:28.699424 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:38:28.699557 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:28.699622 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 16:38:28.699678 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 16:38:28.705636 15305 raft_consensus.cc:1261] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0 [term 1 FOLLOWER]: Refusing update from remote peer 33d07e286b1e402c8d93bab96a9f7f4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:28.705664 15441 raft_consensus.cc:1261] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e [term 1 FOLLOWER]: Refusing update from remote peer 33d07e286b1e402c8d93bab96a9f7f4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:28.706339 15653 consensus_queue.cc:1035] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7903a45d29fa4156b3644783881840c0" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:38:28.706534 15650 consensus_queue.cc:1035] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d77920ddcbc1490d9762ee1165e5a54e" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 44753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 16:38:28.711273 15626 tablet.cc:2369] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:38:28.715816 15656 mvcc.cc:205] Tried to move back new op lower bound from 7334352931650785280 to 7334352931456684032. Current Snapshot: MvccSnapshot[applied={T|T < 7334352931650785280 or (T in {7334352931650785280})}]
I20260928 16:38:28.715529 15657 mvcc.cc:205] Tried to move back new op lower bound from 7334352931650785280 to 7334352931456684032. Current Snapshot: MvccSnapshot[applied={T|T < 7334352931650785280 or (T in {7334352931650785280})}]
I20260928 16:38:28.721539 15157 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:38:28.721668 15157 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:38:28.724040 15157 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0315993ac8d7479f953de60e5e622bf6: Sending DeleteTablet for 3 replicas of tablet 33527dc65c624b818d9623ada1d97623
I20260928 16:38:28.724856 15558 tablet_service.cc:1514] Processing DeleteTablet for tablet 33527dc65c624b818d9623ada1d97623 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:28 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:36800
I20260928 16:38:28.724939 15285 tablet_service.cc:1514] Processing DeleteTablet for tablet 33527dc65c624b818d9623ada1d97623 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:28 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:38126
I20260928 16:38:28.724952 15421 tablet_service.cc:1514] Processing DeleteTablet for tablet 33527dc65c624b818d9623ada1d97623 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:28 UTC) from {username='kudu', principal='kudu/127.27.122.254@KRBTEST.COM'} at 127.0.0.1:58362
I20260928 16:38:28.725339 15667 tablet_replica.cc:330] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0: stopping tablet replica
I20260928 16:38:28.725513 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 15221
I20260928 16:38:28.725658 15667 raft_consensus.cc:2229] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:38:28.725931 15667 raft_consensus.cc:2258] T 33527dc65c624b818d9623ada1d97623 P 7903a45d29fa4156b3644783881840c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:28.726148 15668 tablet_replica.cc:330] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c: stopping tablet replica
I20260928 16:38:28.726410 15668 raft_consensus.cc:2229] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:38:28.726760 15668 raft_consensus.cc:2258] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:28.727630 15669 tablet_replica.cc:330] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e: stopping tablet replica
I20260928 16:38:28.727916 15669 raft_consensus.cc:2229] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:38:28.728222 15668 ts_tablet_manager.cc:1905] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:38:28.728260 15669 raft_consensus.cc:2258] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:28.729640 15669 ts_tablet_manager.cc:1905] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:38:28.731761 15668 ts_tablet_manager.cc:1918] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:38:28.731912 15668 log.cc:1196] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/33527dc65c624b818d9623ada1d97623
I20260928 16:38:28.732300 15669 ts_tablet_manager.cc:1918] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:38:28.732326 15668 ts_tablet_manager.cc:1939] T 33527dc65c624b818d9623ada1d97623 P 33d07e286b1e402c8d93bab96a9f7f4c: Deleting consensus metadata
I20260928 16:38:28.732473 15669 log.cc:1196] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/33527dc65c624b818d9623ada1d97623
I20260928 16:38:28.732852 15669 ts_tablet_manager.cc:1939] T 33527dc65c624b818d9623ada1d97623 P d77920ddcbc1490d9762ee1165e5a54e: Deleting consensus metadata
I20260928 16:38:28.733757 15142 catalog_manager.cc:4841] TS 33d07e286b1e402c8d93bab96a9f7f4c (127.27.122.195:42275): tablet 33527dc65c624b818d9623ada1d97623 (table test-table [id=0fe008c5919841f1bf279be4066f3c9a]) successfully deleted
I20260928 16:38:28.734014 15143 catalog_manager.cc:4841] TS d77920ddcbc1490d9762ee1165e5a54e (127.27.122.194:44753): tablet 33527dc65c624b818d9623ada1d97623 (table test-table [id=0fe008c5919841f1bf279be4066f3c9a]) successfully deleted
W20260928 16:38:28.734498 15143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:35951 (error 108)
I20260928 16:38:28.734779 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 15358
I20260928 16:38:28.736444 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.736168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:35951 (local address 127.0.0.1:38134)
0928 16:38:28.736306 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:28.736375 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:35951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 16:38:28.736682 15143 catalog_manager.cc:4584] TS 7903a45d29fa4156b3644783881840c0 (127.27.122.193:35951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 33527dc65c624b818d9623ada1d97623: Network error: Client connection negotiation failed: client connection to 127.27.122.193:35951: connect: Connection refused (error 111)
I20260928 16:38:28.740868 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 15494
I20260928 16:38:28.749312 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 15126
2026-09-28T16:38:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2311 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T16:38:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:28Z Disabled control of system clock
I20260928 16:38:28.779224 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:37375
--webserver_interface=127.27.122.254
--webserver_port=0
--builtin_ntp_servers=127.27.122.212:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:37375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:38:28.884505 15674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:28.884747 15674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:28.884780 15674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:28.888682 15674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:28.888731 15674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:28.888746 15674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:28.888762 15674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:28.888777 15674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:28.893311 15674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:37375
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:37375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:28.894335 15674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:28.895640 15674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:28.903831 15679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:28.903887 15683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:28.903975 15674 server_base.cc:1034] running on GCE node
W20260928 16:38:28.903901 15681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:28.904582 15674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:28.905669 15674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:28.906834 15674 hybrid_clock.cc:648] HybridClock initialized: now 1790613508906803 us; error 40 us; skew 500 ppm
I20260928 16:38:28.908751 15674 webserver.cc:460] Webserver started at http://127.27.122.254:42343/ using document root <none> and password file <none>
I20260928 16:38:28.909369 15674 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:28.909461 15674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:28.909679 15674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:28.911656 15674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "846ee193e56a45098f026afd638d7611"
format_stamp: "Formatted at 2026-09-28 16:38:28 on dist-test-slave-g9sh"
server_key: "943c236ca36d2775e3efaf1e04eda04b"
server_key_iv: "6efb48c3a3d2660ebb082b7eb1a23675"
server_key_version: "encryptionkey@0"
I20260928 16:38:28.912207 15674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "846ee193e56a45098f026afd638d7611"
format_stamp: "Formatted at 2026-09-28 16:38:28 on dist-test-slave-g9sh"
server_key: "943c236ca36d2775e3efaf1e04eda04b"
server_key_iv: "6efb48c3a3d2660ebb082b7eb1a23675"
server_key_version: "encryptionkey@0"
I20260928 16:38:28.915743 15674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:38:28.918123 15689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:28.919369 15674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:38:28.919524 15674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "846ee193e56a45098f026afd638d7611"
format_stamp: "Formatted at 2026-09-28 16:38:28 on dist-test-slave-g9sh"
server_key: "943c236ca36d2775e3efaf1e04eda04b"
server_key_iv: "6efb48c3a3d2660ebb082b7eb1a23675"
server_key_version: "encryptionkey@0"
I20260928 16:38:28.919644 15674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:28.954404 15674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:28.955138 15674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:28.955338 15674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:28.962232 15674 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:37375
I20260928 16:38:28.962255 15743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:37375 every 8 connection(s)
I20260928 16:38:28.963173 15674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:28.964486 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 15674
I20260928 16:38:28.964623 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:28.964932 28139 external_mini_cluster.cc:1428] Setting key be16094689470d5fc9c585342ec78a61
I20260928 16:38:28.966337 15744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:28.973080 15744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611: Bootstrap starting.
I20260928 16:38:28.974946 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:28.966286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.27.122.254:37375)
0928 16:38:28.966968 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:28.966988 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:28.967031 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:38:28.967907 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:28.967931 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:28.968409 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:28.968790 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:28.969267 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.970697 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.971605 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:28.972299 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:28.972463 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:28.973301 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:28.973306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:28.973319 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:28.973695 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:28.973748 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:28.973768 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:28.973885 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:28.974035 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 16:38:28.974274 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":153,"threads_started":1}
I20260928 16:38:28.975332 15744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:28.976127 15744 log.cc:824] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:28.978021 15744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611: No bootstrap required, opened a new log
I20260928 16:38:28.981726 15744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846ee193e56a45098f026afd638d7611" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37375 } }
I20260928 16:38:28.981923 15744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:28.981957 15744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846ee193e56a45098f026afd638d7611, State: Initialized, Role: FOLLOWER
I20260928 16:38:28.982299 15744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846ee193e56a45098f026afd638d7611" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37375 } }
I20260928 16:38:28.982448 15744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:28.982517 15744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:28.982620 15744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:28.983876 15744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846ee193e56a45098f026afd638d7611" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37375 } }
I20260928 16:38:28.984269 15744 leader_election.cc:302] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 846ee193e56a45098f026afd638d7611; no voters:
I20260928 16:38:28.984552 15744 leader_election.cc:288] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:28.984673 15749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:28.984874 15749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [term 1 LEADER]: Becoming Leader. State: Replica: 846ee193e56a45098f026afd638d7611, State: Running, Role: LEADER
I20260928 16:38:28.985213 15749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846ee193e56a45098f026afd638d7611" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37375 } }
I20260928 16:38:28.985673 15744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:28.986788 15750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "846ee193e56a45098f026afd638d7611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846ee193e56a45098f026afd638d7611" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37375 } } }
I20260928 16:38:28.986923 15750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:28.987280 15751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 846ee193e56a45098f026afd638d7611. Latest consensus state: current_term: 1 leader_uuid: "846ee193e56a45098f026afd638d7611" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846ee193e56a45098f026afd638d7611" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37375 } } }
I20260928 16:38:28.987345 15758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:28.987910 15751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:28.990459 15758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:28.995499 15758 catalog_manager.cc:1347] Generated new cluster ID: e92bbe0dd0194d81a9398c4048ae859e
I20260928 16:38:28.995558 15758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:29.028376 15758 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:29.029605 15758 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:29.042012 15758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611: Generated new TSK 0
I20260928 16:38:29.042791 15758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:29.049767 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.193:0
--local_ip_for_outbound_sockets=127.27.122.193
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:37375
--builtin_ntp_servers=127.27.122.212:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 16:38:29.161574 15768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:29.161810 15768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:29.161849 15768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:29.165822 15768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:29.165876 15768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:29.165930 15768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.193
I20260928 16:38:29.170589 15768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.122.193
--webserver_port=0
--tserver_master_addrs=127.27.122.254:37375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.193
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:29.171650 15768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:29.173054 15768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:38:29.182360 15768 server_base.cc:1034] running on GCE node
W20260928 16:38:29.182371 15773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:29.182369 15776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:29.182389 15774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:29.183043 15768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:29.183583 15768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:29.184749 15768 hybrid_clock.cc:648] HybridClock initialized: now 1790613509184722 us; error 30 us; skew 500 ppm
I20260928 16:38:29.186703 15768 webserver.cc:460] Webserver started at http://127.27.122.193:44869/ using document root <none> and password file <none>
I20260928 16:38:29.187292 15768 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:29.187353 15768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:29.187573 15768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:29.189589 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/data/instance:
uuid: "3fff13f754db4788a7e6bcd80bc90ef9"
format_stamp: "Formatted at 2026-09-28 16:38:29 on dist-test-slave-g9sh"
server_key: "966467f9fc8db7be35651046bee55f5c"
server_key_iv: "2e75708a3da03326de77deb133076601"
server_key_version: "encryptionkey@0"
I20260928 16:38:29.190124 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance:
uuid: "3fff13f754db4788a7e6bcd80bc90ef9"
format_stamp: "Formatted at 2026-09-28 16:38:29 on dist-test-slave-g9sh"
server_key: "966467f9fc8db7be35651046bee55f5c"
server_key_iv: "2e75708a3da03326de77deb133076601"
server_key_version: "encryptionkey@0"
I20260928 16:38:29.193580 15768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:38:29.195822 15782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:29.197120 15768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260928 16:38:29.197233 15768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/wal
uuid: "3fff13f754db4788a7e6bcd80bc90ef9"
format_stamp: "Formatted at 2026-09-28 16:38:29 on dist-test-slave-g9sh"
server_key: "966467f9fc8db7be35651046bee55f5c"
server_key_iv: "2e75708a3da03326de77deb133076601"
server_key_version: "encryptionkey@0"
I20260928 16:38:29.197327 15768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:29.218909 15768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:29.219476 15768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:29.219669 15768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:29.220191 15768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:29.221083 15768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:29.221128 15768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:29.221186 15768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:29.221216 15768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:29.230676 15768 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.193:35247
I20260928 16:38:29.230752 15897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.193:35247 every 8 connection(s)
I20260928 16:38:29.231561 15768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/data/info.pb
I20260928 16:38:29.236506 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 15768
I20260928 16:38:29.236744 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-0/wal/instance
I20260928 16:38:29.237178 28139 external_mini_cluster.cc:1428] Setting key bc4e4dd3d6a79d941f4f3a6c94cf7576
I20260928 16:38:29.239099 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.194:0
--local_ip_for_outbound_sockets=127.27.122.194
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:37375
--builtin_ntp_servers=127.27.122.212:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:38:29.241269 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:29.233361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.193:46077 (local address 127.27.122.254:37375)
0928 16:38:29.233527 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:29.233533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:29.234535 (+ 1002us) server_negotiation.cc:407] Connection header received
0928 16:38:29.235870 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:29.235873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:29.235924 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:29.236010 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:29.236983 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.237768 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.239046 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.239468 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.239609 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:29.240787 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:29.240790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:29.240792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:29.240803 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:29.240822 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:29.240829 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:29.240888 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:29.241098 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 16:38:29.241155 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 16:38:29.242250 15900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:29.233748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37375 (local address 127.27.122.193:46077)
0928 16:38:29.234375 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:29.234414 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:29.235672 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:29.236164 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:29.236171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:29.236262 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:29.236778 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:29.236792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.237905 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.237908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:29.238917 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:29.238927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.239653 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.239656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:29.239794 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:29.240457 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:38:29.240461 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:29.240531 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:38:29.240617 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:29.240941 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:29.240952 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:29.240959 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 16:38:29.241090 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 16:38:29.241396 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
I20260928 16:38:29.243444 15898 heartbeater.cc:347] Connected to a master server at 127.27.122.254:37375
I20260928 16:38:29.243688 15898 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:29.244194 15898 heartbeater.cc:511] Master 127.27.122.254:37375 requested a full tablet report, sending...
I20260928 16:38:29.245692 15706 ts_manager.cc:194] Registered new tserver with Master: 3fff13f754db4788a7e6bcd80bc90ef9 (127.27.122.193:35247)
I20260928 16:38:29.246820 15706 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.122.193:46077
W20260928 16:38:29.352133 15901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:29.352394 15901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:29.352463 15901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:29.356339 15901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:29.356411 15901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:29.356499 15901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.194
I20260928 16:38:29.361058 15901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.122.194
--webserver_port=0
--tserver_master_addrs=127.27.122.254:37375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.194
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:29.362018 15901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:29.363387 15901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:29.372591 15906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:29.372622 15909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:29.372743 15901 server_base.cc:1034] running on GCE node
W20260928 16:38:29.372603 15907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:29.373286 15901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:29.373872 15901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:29.375056 15901 hybrid_clock.cc:648] HybridClock initialized: now 1790613509375023 us; error 44 us; skew 500 ppm
I20260928 16:38:29.376735 15901 webserver.cc:460] Webserver started at http://127.27.122.194:43247/ using document root <none> and password file <none>
I20260928 16:38:29.377333 15901 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:29.377424 15901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:29.377655 15901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:29.379607 15901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/data/instance:
uuid: "428316cc5501414f880e301ebb7d1bfd"
format_stamp: "Formatted at 2026-09-28 16:38:29 on dist-test-slave-g9sh"
server_key: "99bdd7faf46fcb60d3f85ce4b29193ef"
server_key_iv: "a085fa604712647958ac30c711ff592e"
server_key_version: "encryptionkey@0"
I20260928 16:38:29.380157 15901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance:
uuid: "428316cc5501414f880e301ebb7d1bfd"
format_stamp: "Formatted at 2026-09-28 16:38:29 on dist-test-slave-g9sh"
server_key: "99bdd7faf46fcb60d3f85ce4b29193ef"
server_key_iv: "a085fa604712647958ac30c711ff592e"
server_key_version: "encryptionkey@0"
I20260928 16:38:29.383769 15901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:38:29.386382 15915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:29.387796 15901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 16:38:29.387883 15901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/wal
uuid: "428316cc5501414f880e301ebb7d1bfd"
format_stamp: "Formatted at 2026-09-28 16:38:29 on dist-test-slave-g9sh"
server_key: "99bdd7faf46fcb60d3f85ce4b29193ef"
server_key_iv: "a085fa604712647958ac30c711ff592e"
server_key_version: "encryptionkey@0"
I20260928 16:38:29.387959 15901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:29.413317 15901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:29.413890 15901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:29.414032 15901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:29.414513 15901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:29.415493 15901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:29.415560 15901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:29.415624 15901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:29.415664 15901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:29.427431 15901 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.194:35641
I20260928 16:38:29.427440 16030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.194:35641 every 8 connection(s)
I20260928 16:38:29.428668 15901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/data/info.pb
I20260928 16:38:29.435763 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 15901
I20260928 16:38:29.435909 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/wal/instance
I20260928 16:38:29.436215 28139 external_mini_cluster.cc:1428] Setting key b397fdd0de45e14af9d276ce98bbb9c5
I20260928 16:38:29.438452 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.122.195:0
--local_ip_for_outbound_sockets=127.27.122.195
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:37375
--builtin_ntp_servers=127.27.122.212:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 16:38:29.440806 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:29.430666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:48399 (local address 127.27.122.254:37375)
0928 16:38:29.430811 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:29.430815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:29.432161 (+ 1346us) server_negotiation.cc:407] Connection header received
0928 16:38:29.434282 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:29.434286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:29.434363 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:29.434490 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:29.435806 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.436594 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.438310 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.438705 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.438830 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:29.440207 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:29.440213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:29.440217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:29.440233 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:29.440259 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:29.440273 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:29.440321 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:29.440551 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 16:38:29.440629 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:38:29.442317 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:29.431135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37375 (local address 127.27.122.194:48399)
0928 16:38:29.431961 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:29.432041 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:29.434002 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:29.434737 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:29.434747 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:29.434856 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:29.435609 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:29.435632 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.436775 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.436778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:29.438183 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:29.438197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.438887 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.438894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:29.439151 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:29.439856 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:38:29.439860 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:29.439926 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:38:29.440019 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:29.440389 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:29.440399 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:29.440405 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 16:38:29.441019 (+ 614us) client_negotiation.cc:240] Negotiation successful
0928 16:38:29.441313 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":111,"threads_started":1}
I20260928 16:38:29.443615 16031 heartbeater.cc:347] Connected to a master server at 127.27.122.254:37375
I20260928 16:38:29.443905 16031 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:29.444594 16031 heartbeater.cc:511] Master 127.27.122.254:37375 requested a full tablet report, sending...
I20260928 16:38:29.445914 15706 ts_manager.cc:194] Registered new tserver with Master: 428316cc5501414f880e301ebb7d1bfd (127.27.122.194:35641)
I20260928 16:38:29.446768 15706 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.122.194:48399
W20260928 16:38:29.553859 16034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:29.554095 16034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:29.554131 16034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:29.558020 16034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:29.558065 16034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:29.558117 16034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.122.195
I20260928 16:38:29.562644 16034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.122.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.122.195
--webserver_port=0
--tserver_master_addrs=127.27.122.254:37375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.122.195
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:29.563594 16034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:29.564886 16034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:29.574018 16042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:29.574090 16039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:29.574013 16040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:29.574401 16034 server_base.cc:1034] running on GCE node
I20260928 16:38:29.574790 16034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:29.575387 16034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:29.576581 16034 hybrid_clock.cc:648] HybridClock initialized: now 1790613509576552 us; error 46 us; skew 500 ppm
I20260928 16:38:29.578439 16034 webserver.cc:460] Webserver started at http://127.27.122.195:35815/ using document root <none> and password file <none>
I20260928 16:38:29.579030 16034 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:29.579121 16034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:29.579362 16034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:29.581391 16034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/data/instance:
uuid: "816d4f097a4c435287de7c9d486c5763"
format_stamp: "Formatted at 2026-09-28 16:38:29 on dist-test-slave-g9sh"
server_key: "d3e5ef7eff85cb154e9d5a39997cc2b7"
server_key_iv: "1e4b58231361d239b20ddd45f498a25a"
server_key_version: "encryptionkey@0"
I20260928 16:38:29.581939 16034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance:
uuid: "816d4f097a4c435287de7c9d486c5763"
format_stamp: "Formatted at 2026-09-28 16:38:29 on dist-test-slave-g9sh"
server_key: "d3e5ef7eff85cb154e9d5a39997cc2b7"
server_key_iv: "1e4b58231361d239b20ddd45f498a25a"
server_key_version: "encryptionkey@0"
I20260928 16:38:29.585497 16034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:38:29.587965 16049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:29.589172 16034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:29.589293 16034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/wal
uuid: "816d4f097a4c435287de7c9d486c5763"
format_stamp: "Formatted at 2026-09-28 16:38:29 on dist-test-slave-g9sh"
server_key: "d3e5ef7eff85cb154e9d5a39997cc2b7"
server_key_iv: "1e4b58231361d239b20ddd45f498a25a"
server_key_version: "encryptionkey@0"
I20260928 16:38:29.589412 16034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:29.605307 16034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:29.605881 16034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:29.606067 16034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:29.606585 16034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 16:38:29.607436 16034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 16:38:29.607506 16034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:29.607573 16034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 16:38:29.607623 16034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:29.615958 16034 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.195:38961
I20260928 16:38:29.615999 16164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.195:38961 every 8 connection(s)
I20260928 16:38:29.616820 16034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/data/info.pb
I20260928 16:38:29.625118 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 16034
I20260928 16:38:29.625252 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/wal/instance
I20260928 16:38:29.625627 28139 external_mini_cluster.cc:1428] Setting key f9cfc554d5afe13f64b77013b356e89d
I20260928 16:38:29.625865 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:29.618605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.195:50649 (local address 127.27.122.254:37375)
0928 16:38:29.618825 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:29.618829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:29.619736 (+ 907us) server_negotiation.cc:407] Connection header received
0928 16:38:29.621090 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:29.621093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:29.621137 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:29.621210 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:29.622053 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.622599 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.623749 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.624044 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.624143 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:29.625289 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:29.625296 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:29.625299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:29.625317 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:29.625350 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:29.625360 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:29.625428 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:29.625640 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 16:38:29.625726 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260928 16:38:29.626915 16168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:29.618963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37375 (local address 127.27.122.195:50649)
0928 16:38:29.619595 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:29.619634 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:29.620883 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:29.621371 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:29.621377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:29.621456 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:29.621934 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:29.621947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.622705 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.622707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:29.623617 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:29.623627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.624187 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.624189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:29.624334 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:29.624966 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 16:38:29.624968 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:29.625022 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 16:38:29.625106 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:29.625454 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:29.625464 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:29.625497 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 16:38:29.625632 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 16:38:29.625910 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":98,"threads_started":1}
I20260928 16:38:29.628073 16165 heartbeater.cc:347] Connected to a master server at 127.27.122.254:37375
I20260928 16:38:29.628298 16165 heartbeater.cc:464] Registering TS with master...
I20260928 16:38:29.628793 16165 heartbeater.cc:511] Master 127.27.122.254:37375 requested a full tablet report, sending...
I20260928 16:38:29.629825 15706 ts_manager.cc:194] Registered new tserver with Master: 816d4f097a4c435287de7c9d486c5763 (127.27.122.195:38961)
I20260928 16:38:29.630407 15706 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.122.195:50649
I20260928 16:38:29.639544 28139 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 16:38:29.645637 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:29.641614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47454 (local address 127.27.122.254:37375)
0928 16:38:29.641767 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:29.641773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:29.641935 (+ 162us) server_negotiation.cc:407] Connection header received
0928 16:38:29.642102 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:29.642107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:29.642154 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:29.642258 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:29.642579 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.643098 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.643979 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.644229 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.644348 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:29.645143 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:29.645147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:29.645150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:29.645164 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:29.645194 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:29.645203 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:29.645290 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:29.645428 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 16:38:29.645484 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 16:38:29.648149 15706 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 16:38:29.651098 15706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 16:38:29.665333 16177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:29.659389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.194:35641 (local address 127.0.0.1:55272)
0928 16:38:29.659718 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:29.659770 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:29.659901 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:29.660926 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:29.660933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:29.660959 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:29.661088 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:29.661095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.662595 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.662598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:29.663786 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:29.663795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.664741 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.664745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:29.664860 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:29.664934 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 16:38:29.665058 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 16:38:29.665184 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":29,"thread_start_us":76,"threads_started":1}
I20260928 16:38:29.665380 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:29.659517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.27.122.194:35641)
0928 16:38:29.660391 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:29.660404 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:29.660431 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:38:29.660512 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:29.660518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:29.660742 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:29.660925 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:29.661233 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.662462 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.663904 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.664610 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.664733 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:29.664942 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:29.665102 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 16:38:29.665205 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":115,"threads_started":1}
I20260928 16:38:29.665721 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:29.660087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:38961 (local address 127.0.0.1:47844)
0928 16:38:29.660691 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:29.660706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:29.660788 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:29.661351 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:29.661354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:29.661372 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:29.661465 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:29.661471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.662759 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.662763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:29.664214 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:29.664221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.665320 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.665324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:29.665422 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:29.665457 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 16:38:29.665537 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 16:38:29.665604 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":3,"thread_start_us":103,"threads_started":1}
I20260928 16:38:29.666163 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:29.660488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.27.122.195:38961)
0928 16:38:29.660882 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:29.660894 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:29.660936 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:38:29.661016 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:29.661021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:29.661206 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:29.661357 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:29.661603 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.662653 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.664330 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.665000 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.665106 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:29.665259 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:29.665873 (+ 614us) server_negotiation.cc:299] Negotiation successful
0928 16:38:29.665983 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":88,"threads_started":1}
I20260928 16:38:29.665983 16181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:29.660293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37694 (local address 127.27.122.193:35247)
0928 16:38:29.661263 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:29.661273 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:29.661316 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:38:29.661390 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:29.661396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:29.661598 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:29.661760 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:29.661930 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.662739 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.664209 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.665054 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.665168 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:29.665325 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:29.665732 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 16:38:29.665832 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":91,"threads_started":1}
I20260928 16:38:29.668316 15956 tablet_service.cc:1467] Processing CreateTablet for tablet 5db44abd3d3941828789a9bfb0c58ecd (DEFAULT_TABLE table=test-table [id=9b6fd30a429041c0847a31c22ad170c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:29.668350 16098 tablet_service.cc:1467] Processing CreateTablet for tablet 5db44abd3d3941828789a9bfb0c58ecd (DEFAULT_TABLE table=test-table [id=9b6fd30a429041c0847a31c22ad170c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:29.668807 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:29.660097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:35247 (local address 127.0.0.1:37694)
0928 16:38:29.661012 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:29.661026 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:29.661111 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:29.661732 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:29.661735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:29.661753 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:29.661838 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:29.661844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.662836 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.662839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:29.664079 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:29.664090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:29.665396 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:29.665402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:29.665587 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:29.665607 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 16:38:29.665698 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 16:38:29.665768 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":25,"thread_start_us":97,"threads_started":1}
I20260928 16:38:29.669323 15956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db44abd3d3941828789a9bfb0c58ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:29.669344 16098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db44abd3d3941828789a9bfb0c58ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:29.671516 15832 tablet_service.cc:1467] Processing CreateTablet for tablet 5db44abd3d3941828789a9bfb0c58ecd (DEFAULT_TABLE table=test-table [id=9b6fd30a429041c0847a31c22ad170c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 16:38:29.672447 15832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5db44abd3d3941828789a9bfb0c58ecd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:29.675550 16183 tablet_bootstrap.cc:492] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763: Bootstrap starting.
I20260928 16:38:29.676999 16184 tablet_bootstrap.cc:492] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd: Bootstrap starting.
I20260928 16:38:29.677747 16183 tablet_bootstrap.cc:654] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:29.678462 16183 log.cc:824] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:29.679100 16184 tablet_bootstrap.cc:654] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:29.679908 16184 log.cc:824] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:29.680375 16183 tablet_bootstrap.cc:492] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763: No bootstrap required, opened a new log
I20260928 16:38:29.680572 16183 ts_tablet_manager.cc:1397] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 16:38:29.680974 16186 tablet_bootstrap.cc:492] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9: Bootstrap starting.
I20260928 16:38:29.682184 16184 tablet_bootstrap.cc:492] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd: No bootstrap required, opened a new log
I20260928 16:38:29.682343 16184 ts_tablet_manager.cc:1397] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 16:38:29.683054 16186 tablet_bootstrap.cc:654] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:29.683674 16183 raft_consensus.cc:348] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428316cc5501414f880e301ebb7d1bfd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35641 } } peers { permanent_uuid: "816d4f097a4c435287de7c9d486c5763" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38961 } } peers { permanent_uuid: "3fff13f754db4788a7e6bcd80bc90ef9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35247 } }
I20260928 16:38:29.683858 16183 raft_consensus.cc:374] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:29.683825 16186 log.cc:824] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:29.683889 16183 raft_consensus.cc:729] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816d4f097a4c435287de7c9d486c5763, State: Initialized, Role: FOLLOWER
I20260928 16:38:29.684310 16183 consensus_queue.cc:260] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428316cc5501414f880e301ebb7d1bfd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35641 } } peers { permanent_uuid: "816d4f097a4c435287de7c9d486c5763" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38961 } } peers { permanent_uuid: "3fff13f754db4788a7e6bcd80bc90ef9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35247 } }
I20260928 16:38:29.684954 16165 heartbeater.cc:503] Master 127.27.122.254:37375 was elected leader, sending a full tablet report...
I20260928 16:38:29.685297 16183 ts_tablet_manager.cc:1428] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260928 16:38:29.685684 16186 tablet_bootstrap.cc:492] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9: No bootstrap required, opened a new log
I20260928 16:38:29.685858 16186 ts_tablet_manager.cc:1397] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 16:38:29.686617 16184 raft_consensus.cc:348] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428316cc5501414f880e301ebb7d1bfd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35641 } } peers { permanent_uuid: "816d4f097a4c435287de7c9d486c5763" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38961 } } peers { permanent_uuid: "3fff13f754db4788a7e6bcd80bc90ef9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35247 } }
I20260928 16:38:29.686860 16184 raft_consensus.cc:374] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:29.686956 16184 raft_consensus.cc:729] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 428316cc5501414f880e301ebb7d1bfd, State: Initialized, Role: FOLLOWER
I20260928 16:38:29.687386 16184 consensus_queue.cc:260] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428316cc5501414f880e301ebb7d1bfd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35641 } } peers { permanent_uuid: "816d4f097a4c435287de7c9d486c5763" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38961 } } peers { permanent_uuid: "3fff13f754db4788a7e6bcd80bc90ef9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35247 } }
I20260928 16:38:29.688336 16184 ts_tablet_manager.cc:1428] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260928 16:38:29.688656 16031 heartbeater.cc:503] Master 127.27.122.254:37375 was elected leader, sending a full tablet report...
I20260928 16:38:29.690658 16186 raft_consensus.cc:348] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428316cc5501414f880e301ebb7d1bfd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35641 } } peers { permanent_uuid: "816d4f097a4c435287de7c9d486c5763" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38961 } } peers { permanent_uuid: "3fff13f754db4788a7e6bcd80bc90ef9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35247 } }
I20260928 16:38:29.690919 16186 raft_consensus.cc:374] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:29.691030 16186 raft_consensus.cc:729] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fff13f754db4788a7e6bcd80bc90ef9, State: Initialized, Role: FOLLOWER
I20260928 16:38:29.691447 16186 consensus_queue.cc:260] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428316cc5501414f880e301ebb7d1bfd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35641 } } peers { permanent_uuid: "816d4f097a4c435287de7c9d486c5763" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38961 } } peers { permanent_uuid: "3fff13f754db4788a7e6bcd80bc90ef9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35247 } }
I20260928 16:38:29.692044 15898 heartbeater.cc:503] Master 127.27.122.254:37375 was elected leader, sending a full tablet report...
I20260928 16:38:29.692273 16186 ts_tablet_manager.cc:1428] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260928 16:38:29.732667 15899 tablet.cc:2369] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:38:29.867854 16166 tablet.cc:2369] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 16:38:29.929899 16032 tablet.cc:2369] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:38:30.014962 16190 raft_consensus.cc:482] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 16:38:30.015240 16190 raft_consensus.cc:504] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428316cc5501414f880e301ebb7d1bfd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35641 } } peers { permanent_uuid: "816d4f097a4c435287de7c9d486c5763" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38961 } } peers { permanent_uuid: "3fff13f754db4788a7e6bcd80bc90ef9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35247 } }
I20260928 16:38:30.016430 16190 leader_election.cc:288] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 816d4f097a4c435287de7c9d486c5763 (127.27.122.195:38961), 3fff13f754db4788a7e6bcd80bc90ef9 (127.27.122.193:35247)
I20260928 16:38:30.020360 16181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:30.016937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:54311 (local address 127.27.122.193:35247)
0928 16:38:30.017082 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:30.017089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:30.017275 (+ 186us) server_negotiation.cc:407] Connection header received
0928 16:38:30.017412 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:30.017415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:30.017473 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:30.017563 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:30.017863 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:30.018368 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:30.019428 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:30.019818 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:30.019948 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:30.019997 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:30.020178 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 16:38:30.020247 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:38:30.020514 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:30.016842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.122.194:59123 (local address 127.27.122.195:38961)
0928 16:38:30.017009 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:30.017014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:30.017414 (+ 400us) server_negotiation.cc:407] Connection header received
0928 16:38:30.017618 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:30.017623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:30.017684 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:30.017786 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 16:38:30.018071 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:30.018555 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:30.019558 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:30.019955 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:30.020080 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:30.020126 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:30.020310 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 16:38:30.020381 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:38:30.020532 16193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:30.016728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.195:38961 (local address 127.27.122.194:59123)
0928 16:38:30.017327 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:30.017342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:30.017486 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:30.017815 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:30.017818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:30.017835 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:30.017951 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:30.017956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:30.018668 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:30.018673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:30.019430 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:30.019439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:30.020075 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:30.020078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:30.020188 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:30.020205 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 16:38:30.020293 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:38:30.020360 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":251,"thread_start_us":75,"threads_started":1}
I20260928 16:38:30.020376 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:30.016730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:35247 (local address 127.27.122.194:54311)
0928 16:38:30.017166 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:30.017182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:30.017296 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:30.017610 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:30.017613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:30.017630 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 16:38:30.017746 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:30.017751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:30.018480 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:30.018485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:30.019299 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:30.019308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:30.019941 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:30.019944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:30.020043 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:30.020065 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 16:38:30.020160 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 16:38:30.020228 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":106,"threads_started":1}
I20260928 16:38:30.021195 16118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db44abd3d3941828789a9bfb0c58ecd" candidate_uuid: "428316cc5501414f880e301ebb7d1bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816d4f097a4c435287de7c9d486c5763" is_pre_election: true
I20260928 16:38:30.021495 15852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db44abd3d3941828789a9bfb0c58ecd" candidate_uuid: "428316cc5501414f880e301ebb7d1bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fff13f754db4788a7e6bcd80bc90ef9" is_pre_election: true
I20260928 16:38:30.021545 16118 raft_consensus.cc:2454] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428316cc5501414f880e301ebb7d1bfd in term 0.
I20260928 16:38:30.021759 15852 raft_consensus.cc:2454] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 428316cc5501414f880e301ebb7d1bfd in term 0.
I20260928 16:38:30.022042 15917 leader_election.cc:302] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 428316cc5501414f880e301ebb7d1bfd, 816d4f097a4c435287de7c9d486c5763; no voters:
I20260928 16:38:30.022333 16190 raft_consensus.cc:2781] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 16:38:30.022442 16190 raft_consensus.cc:482] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 16:38:30.022552 16190 raft_consensus.cc:3037] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:30.023640 16190 raft_consensus.cc:504] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428316cc5501414f880e301ebb7d1bfd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35641 } } peers { permanent_uuid: "816d4f097a4c435287de7c9d486c5763" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38961 } } peers { permanent_uuid: "3fff13f754db4788a7e6bcd80bc90ef9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35247 } }
I20260928 16:38:30.023972 16190 leader_election.cc:288] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [CANDIDATE]: Term 1 election: Requested vote from peers 816d4f097a4c435287de7c9d486c5763 (127.27.122.195:38961), 3fff13f754db4788a7e6bcd80bc90ef9 (127.27.122.193:35247)
I20260928 16:38:30.024312 15852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db44abd3d3941828789a9bfb0c58ecd" candidate_uuid: "428316cc5501414f880e301ebb7d1bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fff13f754db4788a7e6bcd80bc90ef9"
I20260928 16:38:30.024416 15852 raft_consensus.cc:3037] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:30.025055 16118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5db44abd3d3941828789a9bfb0c58ecd" candidate_uuid: "428316cc5501414f880e301ebb7d1bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "816d4f097a4c435287de7c9d486c5763"
I20260928 16:38:30.025175 16118 raft_consensus.cc:3037] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:30.025614 15852 raft_consensus.cc:2454] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428316cc5501414f880e301ebb7d1bfd in term 1.
I20260928 16:38:30.025915 15919 leader_election.cc:302] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fff13f754db4788a7e6bcd80bc90ef9, 428316cc5501414f880e301ebb7d1bfd; no voters:
I20260928 16:38:30.026369 16118 raft_consensus.cc:2454] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 428316cc5501414f880e301ebb7d1bfd in term 1.
I20260928 16:38:30.026599 16190 raft_consensus.cc:2781] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:30.026818 16190 raft_consensus.cc:686] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [term 1 LEADER]: Becoming Leader. State: Replica: 428316cc5501414f880e301ebb7d1bfd, State: Running, Role: LEADER
I20260928 16:38:30.027132 16190 consensus_queue.cc:237] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428316cc5501414f880e301ebb7d1bfd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35641 } } peers { permanent_uuid: "816d4f097a4c435287de7c9d486c5763" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38961 } } peers { permanent_uuid: "3fff13f754db4788a7e6bcd80bc90ef9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35247 } }
I20260928 16:38:30.030189 15706 catalog_manager.cc:5495] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 428316cc5501414f880e301ebb7d1bfd (127.27.122.194). New cstate: current_term: 1 leader_uuid: "428316cc5501414f880e301ebb7d1bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "428316cc5501414f880e301ebb7d1bfd" member_type: VOTER last_known_addr { host: "127.27.122.194" port: 35641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "816d4f097a4c435287de7c9d486c5763" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fff13f754db4788a7e6bcd80bc90ef9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35247 } health_report { overall_health: UNKNOWN } } }
I20260928 16:38:30.114001 16178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:30.110178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.27.122.194:35641)
0928 16:38:30.110300 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:30.110307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:30.110323 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:38:30.110533 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:30.110536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:30.110603 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:30.110719 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:38:30.110993 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:30.111556 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:30.112428 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:30.112687 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:30.112797 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:30.113163 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:38:30.113609 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:38:30.113750 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:30.113790 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 16:38:30.113843 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 16:38:30.119840 15852 raft_consensus.cc:1261] T 5db44abd3d3941828789a9bfb0c58ecd P 3fff13f754db4788a7e6bcd80bc90ef9 [term 1 FOLLOWER]: Refusing update from remote peer 428316cc5501414f880e301ebb7d1bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:30.119891 16118 raft_consensus.cc:1261] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763 [term 1 FOLLOWER]: Refusing update from remote peer 428316cc5501414f880e301ebb7d1bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:30.120604 16190 consensus_queue.cc:1035] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fff13f754db4788a7e6bcd80bc90ef9" member_type: VOTER last_known_addr { host: "127.27.122.193" port: 35247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:38:30.120797 16194 consensus_queue.cc:1035] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "816d4f097a4c435287de7c9d486c5763" member_type: VOTER last_known_addr { host: "127.27.122.195" port: 38961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:38:30.128702 16199 mvcc.cc:205] Tried to move back new op lower bound from 7334352937443405824 to 7334352937073500160. Current Snapshot: MvccSnapshot[applied={T|T < 7334352937443405824}]
I20260928 16:38:30.136338 15704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:38:30.136508 15704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 16:38:30.139173 15704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 846ee193e56a45098f026afd638d7611: Sending DeleteTablet for 3 replicas of tablet 5db44abd3d3941828789a9bfb0c58ecd
I20260928 16:38:30.139955 15832 tablet_service.cc:1514] Processing DeleteTablet for tablet 5db44abd3d3941828789a9bfb0c58ecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:30 UTC) from {username='slave'} at 127.0.0.1:37694
I20260928 16:38:30.140031 15956 tablet_service.cc:1514] Processing DeleteTablet for tablet 5db44abd3d3941828789a9bfb0c58ecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:30 UTC) from {username='slave'} at 127.0.0.1:55272
I20260928 16:38:30.140471 16098 tablet_service.cc:1514] Processing DeleteTablet for tablet 5db44abd3d3941828789a9bfb0c58ecd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 16:38:30 UTC) from {username='slave'} at 127.0.0.1:47844
I20260928 16:38:30.140858 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 15768
I20260928 16:38:30.145350 16209 tablet_replica.cc:330] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763: stopping tablet replica
I20260928 16:38:30.145702 16209 raft_consensus.cc:2229] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 16:38:30.146034 16209 raft_consensus.cc:2258] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:30.146421 16208 tablet_replica.cc:330] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd: stopping tablet replica
I20260928 16:38:30.146701 16208 raft_consensus.cc:2229] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [term 1 LEADER]: Raft consensus shutting down.
I20260928 16:38:30.147058 16208 raft_consensus.cc:2258] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 16:38:30.147902 16209 ts_tablet_manager.cc:1905] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:38:30.148448 16208 ts_tablet_manager.cc:1905] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 16:38:30.151198 16208 ts_tablet_manager.cc:1918] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 16:38:30.151315 16208 log.cc:1196] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-1/wal/wals/5db44abd3d3941828789a9bfb0c58ecd
I20260928 16:38:30.151654 16208 ts_tablet_manager.cc:1939] T 5db44abd3d3941828789a9bfb0c58ecd P 428316cc5501414f880e301ebb7d1bfd: Deleting consensus metadata
I20260928 16:38:30.151674 16209 ts_tablet_manager.cc:1918] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 16:38:30.151820 15693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.122.193:35247 (error 108)
I20260928 16:38:30.151893 16209 log.cc:1196] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763: Deleting WAL directory at /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790613255639823-28139-0/minicluster-data/ts-2/wal/wals/5db44abd3d3941828789a9bfb0c58ecd
I20260928 16:38:30.152340 16209 ts_tablet_manager.cc:1939] T 5db44abd3d3941828789a9bfb0c58ecd P 816d4f097a4c435287de7c9d486c5763: Deleting consensus metadata
I20260928 16:38:30.152784 15691 catalog_manager.cc:4841] TS 428316cc5501414f880e301ebb7d1bfd (127.27.122.194:35641): tablet 5db44abd3d3941828789a9bfb0c58ecd (table test-table [id=9b6fd30a429041c0847a31c22ad170c2]) successfully deleted
I20260928 16:38:30.153278 16179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:30.152945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.193:35247 (local address 127.0.0.1:37708)
0928 16:38:30.153125 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:30.153195 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.193:35247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260928 16:38:30.153422 15693 catalog_manager.cc:4584] TS 3fff13f754db4788a7e6bcd80bc90ef9 (127.27.122.193:35247): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5db44abd3d3941828789a9bfb0c58ecd: Network error: Client connection negotiation failed: client connection to 127.27.122.193:35247: connect: Connection refused (error 111)
I20260928 16:38:30.153505 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 15901
I20260928 16:38:30.153610 15691 catalog_manager.cc:4841] TS 816d4f097a4c435287de7c9d486c5763 (127.27.122.195:38961): tablet 5db44abd3d3941828789a9bfb0c58ecd (table test-table [id=9b6fd30a429041c0847a31c22ad170c2]) successfully deleted
I20260928 16:38:30.159525 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 16034
I20260928 16:38:30.165345 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 15674
2026-09-28T16:38:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1414 ms)
[----------] 6 tests from EncryptionPolicyTest (10712 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:30 dist-test-slave-g9sh krb5kdc[16213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:30 dist-test-slave-g9sh krb5kdc[16213](info): set up 2 sockets
Sep 28 16:38:30 dist-test-slave-g9sh krb5kdc[16213](info): commencing operation
krb5kdc: starting...
W20260928 16:38:31.207428 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:31 dist-test-slave-g9sh krb5kdc[16213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:38:31.328495 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42173
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:31.437289 16226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:31.437551 16226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:31.437593 16226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:31.441653 16226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:31.441709 16226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:31.441728 16226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:31.441745 16226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:31.441761 16226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:31.446862 16226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42173
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:31.447825 16226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:31.448808 16226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:31.457106 16235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:31.457124 16231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:31.457124 16232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:31.457176 16226 server_base.cc:1034] running on GCE node
I20260928 16:38:31.457860 16226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:31.458909 16226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:31.460134 16226 hybrid_clock.cc:648] HybridClock initialized: now 1790613511460112 us; error 40 us; skew 500 ppm
Sep 28 16:38:31 dist-test-slave-g9sh krb5kdc[16213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:31.463003 16226 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:38:31.463769 16226 webserver.cc:460] Webserver started at http://127.0.0.1:44513/ using document root <none> and password file <none>
I20260928 16:38:31.464286 16226 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:31.464340 16226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:31.464498 16226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:31.466523 16226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "fc2713c6760a479bbe660c4e28b53c65"
format_stamp: "Formatted at 2026-09-28 16:38:31 on dist-test-slave-g9sh"
server_key: "a22c62747e63d1fe0d1b51073aed8527"
server_key_iv: "b6aa791fc00ddb8c2e2f036b1841d881"
server_key_version: "encryptionkey@0"
I20260928 16:38:31.467034 16226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "fc2713c6760a479bbe660c4e28b53c65"
format_stamp: "Formatted at 2026-09-28 16:38:31 on dist-test-slave-g9sh"
server_key: "a22c62747e63d1fe0d1b51073aed8527"
server_key_iv: "b6aa791fc00ddb8c2e2f036b1841d881"
server_key_version: "encryptionkey@0"
I20260928 16:38:31.470587 16226 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:38:31.472862 16242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:31.474105 16226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:38:31.474236 16226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "fc2713c6760a479bbe660c4e28b53c65"
format_stamp: "Formatted at 2026-09-28 16:38:31 on dist-test-slave-g9sh"
server_key: "a22c62747e63d1fe0d1b51073aed8527"
server_key_iv: "b6aa791fc00ddb8c2e2f036b1841d881"
server_key_version: "encryptionkey@0"
I20260928 16:38:31.474354 16226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:31.487253 16226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:31.490195 16226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:31.490343 16226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:31.497996 16226 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42173
I20260928 16:38:31.498014 16296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42173 every 8 connection(s)
I20260928 16:38:31.498909 16226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:31.501654 16297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:31.505177 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 16226
I20260928 16:38:31.505332 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:31.505625 28139 external_mini_cluster.cc:1428] Setting key 8806485e5449fbd427317b2d10c7af0d
I20260928 16:38:31.507309 16297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65: Bootstrap starting.
I20260928 16:38:31.509720 16297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:31.510430 16297 log.cc:824] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:31.512372 16297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65: No bootstrap required, opened a new log
Sep 28 16:38:31 dist-test-slave-g9sh krb5kdc[16213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:38:31.516321 16297 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2713c6760a479bbe660c4e28b53c65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } }
I20260928 16:38:31.516552 16297 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:31.516633 16297 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2713c6760a479bbe660c4e28b53c65, State: Initialized, Role: FOLLOWER
I20260928 16:38:31.517099 16297 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2713c6760a479bbe660c4e28b53c65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } }
I20260928 16:38:31.517243 16297 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:31.517308 16297 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:31.517405 16297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:31.518383 16297 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2713c6760a479bbe660c4e28b53c65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } }
I20260928 16:38:31.518800 16297 leader_election.cc:302] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc2713c6760a479bbe660c4e28b53c65; no voters:
I20260928 16:38:31.519155 16297 leader_election.cc:288] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:31.519249 16302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:31.519440 16302 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [term 1 LEADER]: Becoming Leader. State: Replica: fc2713c6760a479bbe660c4e28b53c65, State: Running, Role: LEADER
I20260928 16:38:31.519824 16302 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2713c6760a479bbe660c4e28b53c65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } }
I20260928 16:38:31.520102 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:31.506918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37233 (local address 127.0.0.1:42173)
0928 16:38:31.507299 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:31.507315 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:31.507348 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:38:31.507991 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:31.508007 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:31.508332 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:31.508604 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:31.509527 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:31.510585 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:31.511679 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:31.512134 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:31.512247 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:31.515088 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:31.515094 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:31.515106 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:31.515131 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:31.516874 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:31.517464 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:31.517469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:31.517474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:31.517607 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:31.517973 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:31.517975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:31.517976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:31.518333 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:31.518491 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:31.518833 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 16:38:31.519063 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":124,"threads_started":1}
I20260928 16:38:31.520383 16297 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:31.521359 16304 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc2713c6760a479bbe660c4e28b53c65. Latest consensus state: current_term: 1 leader_uuid: "fc2713c6760a479bbe660c4e28b53c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2713c6760a479bbe660c4e28b53c65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } } }
I20260928 16:38:31.521535 16304 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:31.521430 16303 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc2713c6760a479bbe660c4e28b53c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc2713c6760a479bbe660c4e28b53c65" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42173 } } }
I20260928 16:38:31.522351 16303 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:31.523761 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 16:38:31.525672 16317 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:38:31.525748 16317 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:38:31.525847 16311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:31.526592 16311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:31.532276 16311 catalog_manager.cc:1347] Generated new cluster ID: 50a879ab3c5d48b391739aea54f51cc1
I20260928 16:38:31.532346 16311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:31.538030 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:31.526002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42939 (local address 127.0.0.1:42173)
0928 16:38:31.526119 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:31.526123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:31.529312 (+ 3189us) server_negotiation.cc:407] Connection header received
0928 16:38:31.529484 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:31.529487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:31.529527 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:31.529609 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:31.530357 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:31.531159 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:31.533010 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:31.533237 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:31.533347 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:31.534764 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:31.534767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:31.534770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:31.534794 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:31.536257 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:31.536736 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:31.536740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:31.536746 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:31.536816 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:31.537170 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:31.537173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:31.537175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:31.537337 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:31.537485 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:31.537754 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 16:38:31.537849 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 16:38:31.574760 16311 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:31.575732 16311 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:31.584800 16311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc2713c6760a479bbe660c4e28b53c65: Generated new TSK 0
I20260928 16:38:31.585659 16311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:31.602068 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 16226
W20260928 16:38:31.608170 16211 connection.cc:535] client connection to 127.0.0.1:42173 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42173): Connection reset by peer (error 104)
2026-09-28T16:38:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:31 dist-test-slave-g9sh krb5kdc[16329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:31 dist-test-slave-g9sh krb5kdc[16329](info): set up 2 sockets
Sep 28 16:38:31 dist-test-slave-g9sh krb5kdc[16329](info): commencing operation
krb5kdc: starting...
W20260928 16:38:32.635526 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:32 dist-test-slave-g9sh krb5kdc[16329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:38:32.754180 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:32.861534 16342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:32.861769 16342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:32.861809 16342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:32.865692 16342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:32.865738 16342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:32.865757 16342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:32.865773 16342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:32.865820 16342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:32.870790 16342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:32.871771 16342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:32.872726 16342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:32.880853 16351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:32.880889 16348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:32.880955 16349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:32.881099 16342 server_base.cc:1034] running on GCE node
I20260928 16:38:32.881748 16342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:32.882783 16342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:32.883972 16342 hybrid_clock.cc:648] HybridClock initialized: now 1790613512883961 us; error 27 us; skew 500 ppm
Sep 28 16:38:32 dist-test-slave-g9sh krb5kdc[16329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:32.886656 16342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:38:32.887513 16342 webserver.cc:460] Webserver started at http://127.0.0.1:44937/ using document root <none> and password file <none>
I20260928 16:38:32.888062 16342 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:32.888156 16342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:32.888388 16342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:32.890431 16342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "80f39aaa11e94427af8c3635c2facee9"
format_stamp: "Formatted at 2026-09-28 16:38:32 on dist-test-slave-g9sh"
server_key: "f38c08b8f8d02ce2a844fd26a7812de3"
server_key_iv: "d53e5488275890c765bb69ba76181058"
server_key_version: "encryptionkey@0"
I20260928 16:38:32.890983 16342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "80f39aaa11e94427af8c3635c2facee9"
format_stamp: "Formatted at 2026-09-28 16:38:32 on dist-test-slave-g9sh"
server_key: "f38c08b8f8d02ce2a844fd26a7812de3"
server_key_iv: "d53e5488275890c765bb69ba76181058"
server_key_version: "encryptionkey@0"
I20260928 16:38:32.894505 16342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 16:38:32.896759 16358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:32.897980 16342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:38:32.898123 16342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "80f39aaa11e94427af8c3635c2facee9"
format_stamp: "Formatted at 2026-09-28 16:38:32 on dist-test-slave-g9sh"
server_key: "f38c08b8f8d02ce2a844fd26a7812de3"
server_key_iv: "d53e5488275890c765bb69ba76181058"
server_key_version: "encryptionkey@0"
I20260928 16:38:32.898229 16342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:32.912381 16342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:32.915446 16342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:32.915632 16342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:32.922901 16342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42267
I20260928 16:38:32.922928 16410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42267 every 8 connection(s)
I20260928 16:38:32.923774 16342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:32.926478 16411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:32.929386 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 16342
I20260928 16:38:32.929551 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:32.929832 28139 external_mini_cluster.cc:1428] Setting key d9a62292d2fa06c8826ed70c8dab07c9
I20260928 16:38:32.932106 16411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9: Bootstrap starting.
I20260928 16:38:32.934525 16411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:32.935396 16411 log.cc:824] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:32.937328 16411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9: No bootstrap required, opened a new log
I20260928 16:38:32.938510 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:32.931085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46991 (local address 127.0.0.1:42267)
0928 16:38:32.931470 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:32.931485 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:32.931524 (+ 39us) server_negotiation.cc:407] Connection header received
0928 16:38:32.932113 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:32.932129 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:32.932494 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:32.932769 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:32.933148 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:32.934095 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:32.935312 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:32.935814 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:32.935979 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:32.936883 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:32.936887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:32.936938 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:32.937317 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:32.937367 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:32.937389 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:32.937543 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:32.937714 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 16:38:32.937851 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":124,"threads_started":1}
I20260928 16:38:32.941671 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:32.941530 16411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f39aaa11e94427af8c3635c2facee9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42267 } }
I20260928 16:38:32.941814 16411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:32.941866 16411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f39aaa11e94427af8c3635c2facee9, State: Initialized, Role: FOLLOWER
I20260928 16:38:32.942318 16411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f39aaa11e94427af8c3635c2facee9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42267 } }
I20260928 16:38:32.942468 16411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:32.942544 16411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:32.942656 16411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:32.943802 16411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f39aaa11e94427af8c3635c2facee9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42267 } }
I20260928 16:38:32.944159 16411 leader_election.cc:302] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80f39aaa11e94427af8c3635c2facee9; no voters:
I20260928 16:38:32.944494 16411 leader_election.cc:288] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:32.944592 16423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:32.944846 16423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [term 1 LEADER]: Becoming Leader. State: Replica: 80f39aaa11e94427af8c3635c2facee9, State: Running, Role: LEADER
I20260928 16:38:32.945210 16423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f39aaa11e94427af8c3635c2facee9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42267 } }
I20260928 16:38:32.945850 16411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:32.946334 16423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80f39aaa11e94427af8c3635c2facee9. Latest consensus state: current_term: 1 leader_uuid: "80f39aaa11e94427af8c3635c2facee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f39aaa11e94427af8c3635c2facee9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42267 } } }
I20260928 16:38:32.947036 16423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:32.946918 16424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80f39aaa11e94427af8c3635c2facee9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f39aaa11e94427af8c3635c2facee9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42267 } } }
I20260928 16:38:32.947350 16424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:32.948846 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:32.943944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36737 (local address 127.0.0.1:42267)
0928 16:38:32.944045 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:32.944049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:32.944227 (+ 178us) server_negotiation.cc:407] Connection header received
0928 16:38:32.944329 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:32.944332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:32.944375 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:32.944467 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:32.944814 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:32.945566 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:32.946731 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:32.947121 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:32.947234 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:32.948298 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:32.948302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:32.948305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:32.948316 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:32.948373 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:32.948390 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:32.948486 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:32.948683 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 16:38:32.948719 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 16:38:32.951210 16439 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:38:32.951288 16439 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:38:32.951371 16432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:32.952198 16432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:32.957588 16432 catalog_manager.cc:1347] Generated new cluster ID: f57266f2145b4319884697b64849a18c
I20260928 16:38:32.957660 16432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:32.967031 16432 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:32.967888 16432 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:32.975387 16432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80f39aaa11e94427af8c3635c2facee9: Generated new TSK 0
I20260928 16:38:32.976078 16432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:32.985212 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 16342
W20260928 16:38:32.991037 16327 connection.cc:535] client connection to 127.0.0.1:42267 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42267): Connection reset by peer (error 104)
2026-09-28T16:38:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:33 dist-test-slave-g9sh krb5kdc[16444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:33 dist-test-slave-g9sh krb5kdc[16444](info): set up 2 sockets
Sep 28 16:38:33 dist-test-slave-g9sh krb5kdc[16444](info): commencing operation
krb5kdc: starting...
W20260928 16:38:34.019430 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:34 dist-test-slave-g9sh krb5kdc[16444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:38:34.140311 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:34.251022 16457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:34.251279 16457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:34.251320 16457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:34.255237 16457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:34.255287 16457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:34.255307 16457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:34.255323 16457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:34.255338 16457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:34.260316 16457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:34.261260 16457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:34.262221 16457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:34.270467 16462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:34.270479 16465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:34.270479 16463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:34.270660 16457 server_base.cc:1034] running on GCE node
I20260928 16:38:34.271288 16457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:34.272277 16457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:34.273465 16457 hybrid_clock.cc:648] HybridClock initialized: now 1790613514273431 us; error 45 us; skew 500 ppm
Sep 28 16:38:34 dist-test-slave-g9sh krb5kdc[16444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613514, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:34.276317 16457 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:38:34.277287 16457 webserver.cc:460] Webserver started at http://127.0.0.1:36973/ using document root <none> and password file <none>
I20260928 16:38:34.277833 16457 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:34.277925 16457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:34.278172 16457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:34.280181 16457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "4a20bad1824045eab1dec26a80de7154"
format_stamp: "Formatted at 2026-09-28 16:38:34 on dist-test-slave-g9sh"
server_key: "54c92a1302b777f5a3fc3ba704600984"
server_key_iv: "a5426d9ad7ea1d7da4187f0f2d39a3ea"
server_key_version: "encryptionkey@0"
I20260928 16:38:34.280735 16457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "4a20bad1824045eab1dec26a80de7154"
format_stamp: "Formatted at 2026-09-28 16:38:34 on dist-test-slave-g9sh"
server_key: "54c92a1302b777f5a3fc3ba704600984"
server_key_iv: "a5426d9ad7ea1d7da4187f0f2d39a3ea"
server_key_version: "encryptionkey@0"
I20260928 16:38:34.284421 16457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 16:38:34.286830 16472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:34.288017 16457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 16:38:34.288167 16457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "4a20bad1824045eab1dec26a80de7154"
format_stamp: "Formatted at 2026-09-28 16:38:34 on dist-test-slave-g9sh"
server_key: "54c92a1302b777f5a3fc3ba704600984"
server_key_iv: "a5426d9ad7ea1d7da4187f0f2d39a3ea"
server_key_version: "encryptionkey@0"
I20260928 16:38:34.288275 16457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:34.292870 16457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:34.295871 16457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:34.296072 16457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:34.303200 16457 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34781
I20260928 16:38:34.303224 16524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34781 every 8 connection(s)
I20260928 16:38:34.304087 16457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:34.306442 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 16457
I20260928 16:38:34.306612 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:34.306934 28139 external_mini_cluster.cc:1428] Setting key 7ee30039289d5ddf89d6118d2e4a23ae
I20260928 16:38:34.307049 16525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:34.312117 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:34.308411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50129 (local address 127.0.0.1:34781)
0928 16:38:34.308847 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:34.308862 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:34.308896 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:38:34.309593 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:34.309609 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:34.309874 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:34.310207 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:34.310986 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:34.310991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:34.311002 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:34.311013 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:34.311072 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:34.311091 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:34.311236 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:34.311308 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 16:38:34.311458 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":134,"threads_started":1}
I20260928 16:38:34.313892 16525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154: Bootstrap starting.
I20260928 16:38:34.314841 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:34.316190 16525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:34.317099 16525 log.cc:824] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:34.318533 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:34.316830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35713 (local address 127.0.0.1:34781)
0928 16:38:34.316951 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:34.316955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:34.317219 (+ 264us) server_negotiation.cc:407] Connection header received
0928 16:38:34.317355 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:34.317357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:34.317384 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:34.317441 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:34.318185 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:34.318189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:34.318192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:34.318204 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:34.318232 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:34.318245 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:34.318326 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:34.318391 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 16:38:34.318423 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 16:38:34.319296 16525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154: No bootstrap required, opened a new log
I20260928 16:38:34.322851 16525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a20bad1824045eab1dec26a80de7154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34781 } }
I20260928 16:38:34.323108 16525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:34.323225 16525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a20bad1824045eab1dec26a80de7154, State: Initialized, Role: FOLLOWER
I20260928 16:38:34.323655 16525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a20bad1824045eab1dec26a80de7154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34781 } }
I20260928 16:38:34.323804 16525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:34.323892 16525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:34.324010 16525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:34.325011 16525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a20bad1824045eab1dec26a80de7154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34781 } }
I20260928 16:38:34.325367 16525 leader_election.cc:302] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a20bad1824045eab1dec26a80de7154; no voters:
I20260928 16:38:34.325691 16525 leader_election.cc:288] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:34.325759 16535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:34.326023 16535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [term 1 LEADER]: Becoming Leader. State: Replica: 4a20bad1824045eab1dec26a80de7154, State: Running, Role: LEADER
I20260928 16:38:34.326357 16535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a20bad1824045eab1dec26a80de7154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34781 } }
I20260928 16:38:34.326885 16525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:34.327975 16537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a20bad1824045eab1dec26a80de7154. Latest consensus state: current_term: 1 leader_uuid: "4a20bad1824045eab1dec26a80de7154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a20bad1824045eab1dec26a80de7154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34781 } } }
I20260928 16:38:34.328104 16537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:34.328428 16536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a20bad1824045eab1dec26a80de7154" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a20bad1824045eab1dec26a80de7154" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34781 } } }
I20260928 16:38:34.328514 16544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:34.329934 16536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:34.331772 16544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:34.337146 16544 catalog_manager.cc:1347] Generated new cluster ID: 1ba260c10941484b874b2a66812e33b9
I20260928 16:38:34.337205 16544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:34.355069 16544 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:34.355372 16544 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:34.366842 16544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a20bad1824045eab1dec26a80de7154: Generated new TSK 0
I20260928 16:38:34.367517 16544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:34.372886 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 16457
2026-09-28T16:38:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:34 dist-test-slave-g9sh krb5kdc[16557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:34 dist-test-slave-g9sh krb5kdc[16557](info): set up 2 sockets
Sep 28 16:38:34 dist-test-slave-g9sh krb5kdc[16557](info): commencing operation
krb5kdc: starting...
W20260928 16:38:35.419719 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:35 dist-test-slave-g9sh krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:38:35.541956 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:35.652050 16571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:35.652341 16571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:35.652501 16571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:35.656437 16571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:35.656526 16571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:35.656563 16571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:35.656617 16571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:35.656653 16571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:35.661850 16571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:35.662861 16571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:35.663890 16571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:38:35.672444 16571 server_base.cc:1034] running on GCE node
W20260928 16:38:35.672402 16576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:35.672410 16577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:35.672676 16579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:35.673291 16571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:35.674264 16571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:35.675420 16571 hybrid_clock.cc:648] HybridClock initialized: now 1790613515675409 us; error 23 us; skew 500 ppm
Sep 28 16:38:35 dist-test-slave-g9sh krb5kdc[16557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613515, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:35.678153 16571 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:38:35.679083 16571 webserver.cc:460] Webserver started at http://127.0.0.1:38745/ using document root <none> and password file <none>
I20260928 16:38:35.679646 16571 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:35.679705 16571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:35.679939 16571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:35.682009 16571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "0492942df116440b868ea5bed17f3515"
format_stamp: "Formatted at 2026-09-28 16:38:35 on dist-test-slave-g9sh"
server_key: "e43547b1c0dcd35b9d137a48b5e3eb33"
server_key_iv: "c58ab5e87b9793dfb8a55e84773863a4"
server_key_version: "encryptionkey@0"
I20260928 16:38:35.682564 16571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "0492942df116440b868ea5bed17f3515"
format_stamp: "Formatted at 2026-09-28 16:38:35 on dist-test-slave-g9sh"
server_key: "e43547b1c0dcd35b9d137a48b5e3eb33"
server_key_iv: "c58ab5e87b9793dfb8a55e84773863a4"
server_key_version: "encryptionkey@0"
I20260928 16:38:35.686136 16571 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:38:35.688508 16586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:35.689764 16571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:35.689901 16571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "0492942df116440b868ea5bed17f3515"
format_stamp: "Formatted at 2026-09-28 16:38:35 on dist-test-slave-g9sh"
server_key: "e43547b1c0dcd35b9d137a48b5e3eb33"
server_key_iv: "c58ab5e87b9793dfb8a55e84773863a4"
server_key_version: "encryptionkey@0"
I20260928 16:38:35.690007 16571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:35.700587 16571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:35.703235 16571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:35.703419 16571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:35.710402 16571 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34885
I20260928 16:38:35.710422 16638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34885 every 8 connection(s)
I20260928 16:38:35.711282 16571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:35.713907 16639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:35.717942 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 16571
I20260928 16:38:35.718091 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:35.718364 28139 external_mini_cluster.cc:1428] Setting key ce1f6d9beaf6f971b73950629fc9c119
I20260928 16:38:35.719617 16639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515: Bootstrap starting.
I20260928 16:38:35.722337 16639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:35.723271 16639 log.cc:824] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:35.725304 16639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515: No bootstrap required, opened a new log
Sep 28 16:38:35 dist-test-slave-g9sh krb5kdc[16557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:38:35.729578 16639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0492942df116440b868ea5bed17f3515" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34885 } }
I20260928 16:38:35.729849 16639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:35.729919 16639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0492942df116440b868ea5bed17f3515, State: Initialized, Role: FOLLOWER
I20260928 16:38:35.730417 16639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0492942df116440b868ea5bed17f3515" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34885 } }
I20260928 16:38:35.730571 16639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:35.730657 16639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:35.730758 16639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:35.731894 16639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0492942df116440b868ea5bed17f3515" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34885 } }
I20260928 16:38:35.732242 16639 leader_election.cc:302] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0492942df116440b868ea5bed17f3515; no voters:
I20260928 16:38:35.732643 16639 leader_election.cc:288] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:35.732730 16644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:35.732980 16644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [term 1 LEADER]: Becoming Leader. State: Replica: 0492942df116440b868ea5bed17f3515, State: Running, Role: LEADER
I20260928 16:38:35.733299 16644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0492942df116440b868ea5bed17f3515" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34885 } }
I20260928 16:38:35.733525 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:35.719752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56609 (local address 127.0.0.1:34885)
0928 16:38:35.720175 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:35.720195 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:35.720236 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:38:35.721109 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:35.721132 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:35.721541 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:35.721894 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:35.722643 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:35.724032 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:35.724945 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:35.725622 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:35.725791 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:35.728267 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:35.728273 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:35.728291 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:35.728322 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:35.730318 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:35.730822 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:35.730828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:35.730835 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:35.730981 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:35.731238 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:35.731241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:35.731243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:35.731746 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:35.731939 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:35.732227 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 16:38:35.732528 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":96,"threads_started":1}
I20260928 16:38:35.733982 16639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:35.734481 16644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0492942df116440b868ea5bed17f3515. Latest consensus state: current_term: 1 leader_uuid: "0492942df116440b868ea5bed17f3515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0492942df116440b868ea5bed17f3515" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34885 } } }
I20260928 16:38:35.734592 16644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:35.734803 16644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0492942df116440b868ea5bed17f3515" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0492942df116440b868ea5bed17f3515" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34885 } } }
I20260928 16:38:35.734905 16644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:35.735653 16648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:35.736795 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:35.739228 16648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:35.745376 16648 catalog_manager.cc:1347] Generated new cluster ID: bc737ff3eb304b69b472eff410e36e8f
I20260928 16:38:35.745436 16648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:35.749615 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:35.741345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44389 (local address 127.0.0.1:34885)
0928 16:38:35.741449 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:35.741453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:35.741715 (+ 262us) server_negotiation.cc:407] Connection header received
0928 16:38:35.741838 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:35.741841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:35.741887 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:35.741965 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:35.742797 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:35.743415 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:35.744602 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:35.744948 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:35.745049 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:35.746499 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:35.746503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:35.746506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:35.746534 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:35.747948 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:35.748368 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:35.748372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:35.748374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:35.748451 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:35.748731 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:35.748736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:35.748737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:35.748949 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:35.749093 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:35.749327 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 16:38:35.749442 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 16:38:35.755788 16648 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:35.756628 16648 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:35.762291 16648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0492942df116440b868ea5bed17f3515: Generated new TSK 0
I20260928 16:38:35.763005 16648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:35.774353 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 16571
2026-09-28T16:38:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:35 dist-test-slave-g9sh krb5kdc[16670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:35 dist-test-slave-g9sh krb5kdc[16670](info): set up 2 sockets
Sep 28 16:38:35 dist-test-slave-g9sh krb5kdc[16670](info): commencing operation
krb5kdc: starting...
W20260928 16:38:36.819305 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:36 dist-test-slave-g9sh krb5kdc[16670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:38:36.939963 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:37.047199 16683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:37.047493 16683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:37.047580 16683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:37.051524 16683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:37.051611 16683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:37.051651 16683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:37.051708 16683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:37.051748 16683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:37.057348 16683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:37.058372 16683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:37.059406 16683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:37.067531 16690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:37.067575 16689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:37.067575 16692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:37.067884 16683 server_base.cc:1034] running on GCE node
I20260928 16:38:37.068385 16683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:37.069496 16683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:37.070703 16683 hybrid_clock.cc:648] HybridClock initialized: now 1790613517070673 us; error 41 us; skew 500 ppm
Sep 28 16:38:37 dist-test-slave-g9sh krb5kdc[16670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:37.073594 16683 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:38:37.074424 16683 webserver.cc:460] Webserver started at http://127.0.0.1:34405/ using document root <none> and password file <none>
I20260928 16:38:37.074967 16683 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:37.075055 16683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:37.075278 16683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:37.077317 16683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "84b461cbdede4a4ca9b44aaedbac8d91"
format_stamp: "Formatted at 2026-09-28 16:38:37 on dist-test-slave-g9sh"
server_key: "2dd0c678dada4d7040125e48c9967554"
server_key_iv: "f083bebc5da9b0fdb19e40acc67e1332"
server_key_version: "encryptionkey@0"
I20260928 16:38:37.077862 16683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "84b461cbdede4a4ca9b44aaedbac8d91"
format_stamp: "Formatted at 2026-09-28 16:38:37 on dist-test-slave-g9sh"
server_key: "2dd0c678dada4d7040125e48c9967554"
server_key_iv: "f083bebc5da9b0fdb19e40acc67e1332"
server_key_version: "encryptionkey@0"
I20260928 16:38:37.081372 16683 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:38:37.083700 16699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:37.084954 16683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:37.085119 16683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "84b461cbdede4a4ca9b44aaedbac8d91"
format_stamp: "Formatted at 2026-09-28 16:38:37 on dist-test-slave-g9sh"
server_key: "2dd0c678dada4d7040125e48c9967554"
server_key_iv: "f083bebc5da9b0fdb19e40acc67e1332"
server_key_version: "encryptionkey@0"
I20260928 16:38:37.085224 16683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:37.122772 16683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:37.126256 16683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:37.126523 16683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:37.133812 16683 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42685
I20260928 16:38:37.133839 16752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42685 every 8 connection(s)
I20260928 16:38:37.134770 16683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:37.135593 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 16683
I20260928 16:38:37.135751 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:37.136061 28139 external_mini_cluster.cc:1428] Setting key 07faec52f0f0675a6a387462e3bc5f7e
I20260928 16:38:37.138070 16753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:37.144965 16753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91: Bootstrap starting.
I20260928 16:38:37.146426 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:37.137524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52577 (local address 127.0.0.1:42685)
0928 16:38:37.137950 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:37.137968 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:37.138011 (+ 43us) server_negotiation.cc:407] Connection header received
0928 16:38:37.138853 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:37.138876 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:37.139318 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:37.139695 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:37.140087 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:37.141686 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:37.142623 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:37.143329 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:37.143490 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:37.144404 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:37.144410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:37.144427 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:37.144940 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:37.145010 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:37.145046 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:37.145141 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:37.145300 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 16:38:37.145505 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":111,"threads_started":1}
I20260928 16:38:37.147568 16753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:37.148458 16753 log.cc:824] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:37.149595 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:37.150583 16753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91: No bootstrap required, opened a new log
I20260928 16:38:37.155143 16753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b461cbdede4a4ca9b44aaedbac8d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42685 } }
I20260928 16:38:37.155457 16753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:37.155534 16753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b461cbdede4a4ca9b44aaedbac8d91, State: Initialized, Role: FOLLOWER
I20260928 16:38:37.155827 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:37.151586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49005 (local address 127.0.0.1:42685)
0928 16:38:37.151712 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:37.151717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:37.151888 (+ 171us) server_negotiation.cc:407] Connection header received
0928 16:38:37.152039 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:37.152041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:37.152077 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:37.152142 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:37.152448 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:37.153016 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:37.154258 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:37.154474 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:37.154549 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:37.155333 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:37.155335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:37.155338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:37.155348 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:37.155372 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:37.155382 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:37.155476 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:37.155618 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 16:38:37.155668 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 16:38:37.156071 16753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b461cbdede4a4ca9b44aaedbac8d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42685 } }
I20260928 16:38:37.156261 16753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:37.156327 16753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:37.156422 16753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:37.157470 16753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b461cbdede4a4ca9b44aaedbac8d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42685 } }
I20260928 16:38:37.157763 16753 leader_election.cc:302] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84b461cbdede4a4ca9b44aaedbac8d91; no voters:
I20260928 16:38:37.158048 16753 leader_election.cc:288] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:37.158135 16763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:37.158316 16763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [term 1 LEADER]: Becoming Leader. State: Replica: 84b461cbdede4a4ca9b44aaedbac8d91, State: Running, Role: LEADER
I20260928 16:38:37.158667 16763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b461cbdede4a4ca9b44aaedbac8d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42685 } }
I20260928 16:38:37.159380 16753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:37.160732 16764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84b461cbdede4a4ca9b44aaedbac8d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b461cbdede4a4ca9b44aaedbac8d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42685 } } }
I20260928 16:38:37.160744 16765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84b461cbdede4a4ca9b44aaedbac8d91. Latest consensus state: current_term: 1 leader_uuid: "84b461cbdede4a4ca9b44aaedbac8d91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84b461cbdede4a4ca9b44aaedbac8d91" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42685 } } }
I20260928 16:38:37.160945 16765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:37.161371 16764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:37.161473 16774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:37.164181 16774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:37.170229 16774 catalog_manager.cc:1347] Generated new cluster ID: a98216d4f95542cd95ac1d01e11e88b1
I20260928 16:38:37.170318 16774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:37.187891 16774 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:37.189029 16774 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:37.197916 16774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84b461cbdede4a4ca9b44aaedbac8d91: Generated new TSK 0
I20260928 16:38:37.198594 16774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:37.210633 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 16683
2026-09-28T16:38:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:37 dist-test-slave-g9sh krb5kdc[16784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:37 dist-test-slave-g9sh krb5kdc[16784](info): set up 2 sockets
Sep 28 16:38:37 dist-test-slave-g9sh krb5kdc[16784](info): commencing operation
krb5kdc: starting...
W20260928 16:38:38.251458 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:38 dist-test-slave-g9sh krb5kdc[16784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:38:38.369625 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:38.477105 16797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:38.477356 16797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:38.477396 16797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:38.481339 16797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:38.481396 16797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:38.481415 16797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:38.481431 16797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:38.481446 16797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:38.486398 16797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:38.487295 16797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:38.488204 16797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:38.496387 16803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:38.496454 16805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:38.496387 16802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:38.496589 16797 server_base.cc:1034] running on GCE node
I20260928 16:38:38.497273 16797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:38.498267 16797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:38.499430 16797 hybrid_clock.cc:648] HybridClock initialized: now 1790613518499410 us; error 31 us; skew 500 ppm
Sep 28 16:38:38 dist-test-slave-g9sh krb5kdc[16784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613518, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:38.502074 16797 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:38:38.502884 16797 webserver.cc:460] Webserver started at http://127.0.0.1:38995/ using document root <none> and password file <none>
I20260928 16:38:38.503415 16797 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:38.503475 16797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:38.503705 16797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:38.505730 16797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "3fceda5fc867407a8014907f41f18025"
format_stamp: "Formatted at 2026-09-28 16:38:38 on dist-test-slave-g9sh"
server_key: "4cfb23c87af78abe1345c4d9458d73f9"
server_key_iv: "4b1982f229d0644d238fe9f7b78c63af"
server_key_version: "encryptionkey@0"
I20260928 16:38:38.506291 16797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "3fceda5fc867407a8014907f41f18025"
format_stamp: "Formatted at 2026-09-28 16:38:38 on dist-test-slave-g9sh"
server_key: "4cfb23c87af78abe1345c4d9458d73f9"
server_key_iv: "4b1982f229d0644d238fe9f7b78c63af"
server_key_version: "encryptionkey@0"
I20260928 16:38:38.509783 16797 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:38:38.512070 16812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:38.513285 16797 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:38.513427 16797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "3fceda5fc867407a8014907f41f18025"
format_stamp: "Formatted at 2026-09-28 16:38:38 on dist-test-slave-g9sh"
server_key: "4cfb23c87af78abe1345c4d9458d73f9"
server_key_iv: "4b1982f229d0644d238fe9f7b78c63af"
server_key_version: "encryptionkey@0"
I20260928 16:38:38.513532 16797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:38.517820 16797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:38.520663 16797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:38.520856 16797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:38.528443 16797 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36311
I20260928 16:38:38.528455 16866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36311 every 8 connection(s)
I20260928 16:38:38.529510 16797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:38.532104 16867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:38.535256 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 16797
I20260928 16:38:38.535413 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:38.535696 28139 external_mini_cluster.cc:1428] Setting key 66d109e250dda094396feef36fa759d3
I20260928 16:38:38.537894 16867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025: Bootstrap starting.
I20260928 16:38:38.540195 16867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:38.540808 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:38.537022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58389 (local address 127.0.0.1:36311)
0928 16:38:38.537459 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:38.537473 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:38.537506 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:38:38.538192 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:38.538209 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:38.538484 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:38.538819 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:38.539607 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:38.539612 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:38.539623 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:38.539634 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:38.539694 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:38.539712 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:38.539829 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:38.539938 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 16:38:38.540090 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":132,"threads_started":1}
I20260928 16:38:38.540984 16867 log.cc:824] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:38.542935 16867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025: No bootstrap required, opened a new log
I20260928 16:38:38.543519 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:38.546749 16867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fceda5fc867407a8014907f41f18025" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } }
I20260928 16:38:38.546947 16867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:38.547030 16867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fceda5fc867407a8014907f41f18025, State: Initialized, Role: FOLLOWER
I20260928 16:38:38.547508 16867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fceda5fc867407a8014907f41f18025" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } }
I20260928 16:38:38.547688 16867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:38.547768 16867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:38.547884 16867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:38.548651 16870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:38.545500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37631 (local address 127.0.0.1:36311)
0928 16:38:38.545605 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:38.545609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:38.547294 (+ 1685us) server_negotiation.cc:407] Connection header received
0928 16:38:38.547409 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:38.547411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:38.547441 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:38.547504 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:38.548262 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:38.548265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:38.548267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:38.548277 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:38.548300 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:38.548309 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:38.548374 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:38.548478 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 16:38:38.548506 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 16:38:38.549089 16867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fceda5fc867407a8014907f41f18025" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } }
I20260928 16:38:38.549425 16867 leader_election.cc:302] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fceda5fc867407a8014907f41f18025; no voters:
I20260928 16:38:38.549748 16867 leader_election.cc:288] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:38.549819 16877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:38.550073 16877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [term 1 LEADER]: Becoming Leader. State: Replica: 3fceda5fc867407a8014907f41f18025, State: Running, Role: LEADER
I20260928 16:38:38.550398 16877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fceda5fc867407a8014907f41f18025" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } }
I20260928 16:38:38.550936 16867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:38.552042 16878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fceda5fc867407a8014907f41f18025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fceda5fc867407a8014907f41f18025" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } } }
I20260928 16:38:38.552183 16878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:38.552568 16886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:38.552520 16879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fceda5fc867407a8014907f41f18025. Latest consensus state: current_term: 1 leader_uuid: "3fceda5fc867407a8014907f41f18025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fceda5fc867407a8014907f41f18025" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36311 } } }
I20260928 16:38:38.553160 16879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:38.555606 16886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:38.560508 16886 catalog_manager.cc:1347] Generated new cluster ID: ce457185876b4a2f83e2ce9c72508dab
I20260928 16:38:38.560567 16886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:38.571564 16886 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:38.571851 16886 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:38.579056 16886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fceda5fc867407a8014907f41f18025: Generated new TSK 0
I20260928 16:38:38.579710 16886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:38.583494 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 16797
2026-09-28T16:38:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:38 dist-test-slave-g9sh krb5kdc[16898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:38 dist-test-slave-g9sh krb5kdc[16898](info): set up 2 sockets
Sep 28 16:38:38 dist-test-slave-g9sh krb5kdc[16898](info): commencing operation
krb5kdc: starting...
W20260928 16:38:39.635732 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:39 dist-test-slave-g9sh krb5kdc[16898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:38:39.754048 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41405
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:39.862072 16911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:39.862314 16911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:39.862355 16911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:39.866333 16911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:39.866385 16911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:39.866405 16911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:39.866421 16911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:39.866458 16911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:39.871605 16911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:39.872566 16911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:39.873591 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:39.882000 16919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:39.882016 16916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:39.882045 16917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:39.882308 16911 server_base.cc:1034] running on GCE node
I20260928 16:38:39.882858 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:39.883915 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:39.885092 16911 hybrid_clock.cc:648] HybridClock initialized: now 1790613519885062 us; error 42 us; skew 500 ppm
Sep 28 16:38:39 dist-test-slave-g9sh krb5kdc[16898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613519, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:39.887682 16911 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:38:39.888533 16911 webserver.cc:460] Webserver started at http://127.0.0.1:36883/ using document root <none> and password file <none>
I20260928 16:38:39.889169 16911 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:39.889259 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:39.889487 16911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:39.891424 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "4ca988f2ff1b43dcb1ae31bb451f0019"
format_stamp: "Formatted at 2026-09-28 16:38:39 on dist-test-slave-g9sh"
server_key: "c71e313e9cba67999f2fe64061152958"
server_key_iv: "7147e0bded6227e1c722247c741437cc"
server_key_version: "encryptionkey@0"
I20260928 16:38:39.891980 16911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "4ca988f2ff1b43dcb1ae31bb451f0019"
format_stamp: "Formatted at 2026-09-28 16:38:39 on dist-test-slave-g9sh"
server_key: "c71e313e9cba67999f2fe64061152958"
server_key_iv: "7147e0bded6227e1c722247c741437cc"
server_key_version: "encryptionkey@0"
I20260928 16:38:39.895483 16911 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 16:38:39.897789 16926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:39.899127 16911 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:39.899277 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "4ca988f2ff1b43dcb1ae31bb451f0019"
format_stamp: "Formatted at 2026-09-28 16:38:39 on dist-test-slave-g9sh"
server_key: "c71e313e9cba67999f2fe64061152958"
server_key_iv: "7147e0bded6227e1c722247c741437cc"
server_key_version: "encryptionkey@0"
I20260928 16:38:39.899380 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:39.923332 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:39.926218 16911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:39.926416 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:39.933801 16911 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41405
I20260928 16:38:39.933826 16978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41405 every 8 connection(s)
I20260928 16:38:39.934716 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:39.937372 16979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:39.939491 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 16911
I20260928 16:38:39.939622 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:39.939911 28139 external_mini_cluster.cc:1428] Setting key ed341b14b6904db3b505cc6a4b3f0372
I20260928 16:38:39.943812 16979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019: Bootstrap starting.
I20260928 16:38:39.946419 16979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:39.947296 16979 log.cc:824] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:39.949409 16979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019: No bootstrap required, opened a new log
Sep 28 16:38:39 dist-test-slave-g9sh krb5kdc[16898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:38:39.952796 16979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca988f2ff1b43dcb1ae31bb451f0019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41405 } }
I20260928 16:38:39.953127 16979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:39.953193 16979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ca988f2ff1b43dcb1ae31bb451f0019, State: Initialized, Role: FOLLOWER
I20260928 16:38:39.953724 16979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca988f2ff1b43dcb1ae31bb451f0019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41405 } }
I20260928 16:38:39.953898 16979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:39.954003 16979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:39.954118 16979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:39.955096 16979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca988f2ff1b43dcb1ae31bb451f0019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41405 } }
I20260928 16:38:39.955447 16979 leader_election.cc:302] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ca988f2ff1b43dcb1ae31bb451f0019; no voters:
I20260928 16:38:39.955745 16979 leader_election.cc:288] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:39.955854 16984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:39.956053 16984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [term 1 LEADER]: Becoming Leader. State: Replica: 4ca988f2ff1b43dcb1ae31bb451f0019, State: Running, Role: LEADER
I20260928 16:38:39.956157 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:39.941286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:45823 (local address 127.0.0.1:41405)
0928 16:38:39.941710 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:39.941730 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:39.941774 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:38:39.942722 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:39.942747 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:39.943165 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:39.943533 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:39.944423 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:39.945937 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:39.946823 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:39.947529 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:39.947699 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:39.950525 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:39.950530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:39.950542 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:39.950568 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:39.952764 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:39.953381 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:39.953386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:39.953390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:39.953494 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:39.953881 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:39.953885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:39.953887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:39.954466 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:39.954647 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:39.954973 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 16:38:39.955214 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":121,"threads_started":1}
I20260928 16:38:39.956425 16984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca988f2ff1b43dcb1ae31bb451f0019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41405 } }
I20260928 16:38:39.957056 16979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:39.957595 16984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ca988f2ff1b43dcb1ae31bb451f0019. Latest consensus state: current_term: 1 leader_uuid: "4ca988f2ff1b43dcb1ae31bb451f0019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca988f2ff1b43dcb1ae31bb451f0019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41405 } } }
I20260928 16:38:39.957760 16984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:39.958072 16985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ca988f2ff1b43dcb1ae31bb451f0019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca988f2ff1b43dcb1ae31bb451f0019" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41405 } } }
I20260928 16:38:39.958164 16990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:39.958238 16985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:39.959506 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:39.961983 16990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:39.967474 16990 catalog_manager.cc:1347] Generated new cluster ID: f0829b7366354569a63187fd7300e32d
I20260928 16:38:39.967566 16990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:39.972950 16982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:39.961738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:42439 (local address 127.0.0.1:41405)
0928 16:38:39.961835 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:39.961839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:39.962438 (+ 599us) server_negotiation.cc:407] Connection header received
0928 16:38:39.962635 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:39.962638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:39.962693 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:39.962745 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:39.965018 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:39.965678 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:39.966872 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:39.967207 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:39.967312 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:39.969021 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:39.969028 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:39.969032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:39.969061 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:39.970769 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:39.971347 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:39.971350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:39.971352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:39.971436 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:39.971852 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:39.971856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:39.971858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:39.972060 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:39.972198 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:39.972674 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 16:38:39.972773 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 16:38:39.976361 16990 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:39.977252 16990 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:39.987633 16990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ca988f2ff1b43dcb1ae31bb451f0019: Generated new TSK 0
I20260928 16:38:39.988325 16990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:39.996634 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 16911
W20260928 16:38:40.002470 16896 connection.cc:535] client connection to 127.0.0.1:41405 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41405): Connection reset by peer (error 104)
2026-09-28T16:38:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:40 dist-test-slave-g9sh krb5kdc[17010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:40 dist-test-slave-g9sh krb5kdc[17010](info): set up 2 sockets
Sep 28 16:38:40 dist-test-slave-g9sh krb5kdc[17010](info): commencing operation
krb5kdc: starting...
W20260928 16:38:41.035682 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:41 dist-test-slave-g9sh krb5kdc[17010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:38:41.156621 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:41.264554 17024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:41.264844 17024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:41.264956 17024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:41.268841 17024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:41.268950 17024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:41.268999 17024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:41.269039 17024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:41.269074 17024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:41.274137 17024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:41.275153 17024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:41.276168 17024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:38:41.284598 17024 server_base.cc:1034] running on GCE node
W20260928 16:38:41.284578 17030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:41.284583 17029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:41.284785 17032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:41.285374 17024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:41.286424 17024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:41.287588 17024 hybrid_clock.cc:648] HybridClock initialized: now 1790613521287574 us; error 26 us; skew 500 ppm
Sep 28 16:38:41 dist-test-slave-g9sh krb5kdc[17010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:41.290442 17024 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:38:41.291348 17024 webserver.cc:460] Webserver started at http://127.0.0.1:46737/ using document root <none> and password file <none>
I20260928 16:38:41.291903 17024 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:41.291963 17024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:41.292187 17024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:41.294243 17024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "859c159913954436a5b5d9c57c1f5081"
format_stamp: "Formatted at 2026-09-28 16:38:41 on dist-test-slave-g9sh"
server_key: "ac460009b49a10ca5972e64eca0e6e88"
server_key_iv: "af36126bd2f6c901461da8fc3e09a8ea"
server_key_version: "encryptionkey@0"
I20260928 16:38:41.294792 17024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "859c159913954436a5b5d9c57c1f5081"
format_stamp: "Formatted at 2026-09-28 16:38:41 on dist-test-slave-g9sh"
server_key: "ac460009b49a10ca5972e64eca0e6e88"
server_key_iv: "af36126bd2f6c901461da8fc3e09a8ea"
server_key_version: "encryptionkey@0"
I20260928 16:38:41.298305 17024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:38:41.300588 17039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:41.301790 17024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:41.301908 17024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "859c159913954436a5b5d9c57c1f5081"
format_stamp: "Formatted at 2026-09-28 16:38:41 on dist-test-slave-g9sh"
server_key: "ac460009b49a10ca5972e64eca0e6e88"
server_key_iv: "af36126bd2f6c901461da8fc3e09a8ea"
server_key_version: "encryptionkey@0"
I20260928 16:38:41.302034 17024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:41.319744 17024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:41.322504 17024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:41.322695 17024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:41.330696 17024 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39895
I20260928 16:38:41.331025 17093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39895 every 8 connection(s)
I20260928 16:38:41.332006 17024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:41.332609 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 17024
I20260928 16:38:41.332775 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:41.333174 28139 external_mini_cluster.cc:1428] Setting key 866c2a239eb03ae07358cc64e02444a2
I20260928 16:38:41.335640 17094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:41.342361 17094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081: Bootstrap starting.
I20260928 16:38:41.343741 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:41.334572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:49305 (local address 127.0.0.1:39895)
0928 16:38:41.335041 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:41.335063 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:41.335793 (+ 730us) server_negotiation.cc:407] Connection header received
0928 16:38:41.336619 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:41.336642 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:41.337098 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:41.337445 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:41.337841 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:41.339340 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:41.340188 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:41.340873 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:41.341064 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:41.341897 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:41.341903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:41.341918 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:41.342373 (+ 455us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:41.342440 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:41.342470 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:41.342605 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:41.342745 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 16:38:41.342866 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":142,"threads_started":1}
I20260928 16:38:41.344937 17094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:41.345757 17094 log.cc:824] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:41.346580 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:41.347970 17094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081: No bootstrap required, opened a new log
I20260928 16:38:41.352687 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:41.348702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:60727 (local address 127.0.0.1:39895)
0928 16:38:41.348821 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:41.348825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:41.349003 (+ 178us) server_negotiation.cc:407] Connection header received
0928 16:38:41.349132 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:41.349134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:41.349179 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:41.349242 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:41.349511 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:41.349973 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:41.351116 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:41.351324 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:41.351396 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:41.352198 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:41.352201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:41.352202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:41.352213 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:41.352234 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:41.352244 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:41.352306 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:41.352462 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 16:38:41.352511 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:38:41.352515 17094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859c159913954436a5b5d9c57c1f5081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39895 } }
I20260928 16:38:41.352815 17094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:41.352898 17094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859c159913954436a5b5d9c57c1f5081, State: Initialized, Role: FOLLOWER
I20260928 16:38:41.353382 17094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859c159913954436a5b5d9c57c1f5081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39895 } }
I20260928 16:38:41.353538 17094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:41.353606 17094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:41.353741 17094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:41.354566 17094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859c159913954436a5b5d9c57c1f5081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39895 } }
I20260928 16:38:41.354842 17094 leader_election.cc:302] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 859c159913954436a5b5d9c57c1f5081; no voters:
I20260928 16:38:41.355072 17094 leader_election.cc:288] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:41.355180 17104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:41.355427 17104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [term 1 LEADER]: Becoming Leader. State: Replica: 859c159913954436a5b5d9c57c1f5081, State: Running, Role: LEADER
I20260928 16:38:41.355780 17104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859c159913954436a5b5d9c57c1f5081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39895 } }
I20260928 16:38:41.356230 17094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:41.357336 17106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 859c159913954436a5b5d9c57c1f5081. Latest consensus state: current_term: 1 leader_uuid: "859c159913954436a5b5d9c57c1f5081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859c159913954436a5b5d9c57c1f5081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39895 } } }
I20260928 16:38:41.357462 17106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:41.357872 17113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:41.357848 17105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "859c159913954436a5b5d9c57c1f5081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "859c159913954436a5b5d9c57c1f5081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39895 } } }
I20260928 16:38:41.358470 17105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:41.360666 17113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:41.365840 17113 catalog_manager.cc:1347] Generated new cluster ID: caabd7b44ba946799c0393371d321e90
I20260928 16:38:41.365907 17113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:41.378047 17113 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:41.378924 17113 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:41.387585 17113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 859c159913954436a5b5d9c57c1f5081: Generated new TSK 0
I20260928 16:38:41.388222 17113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:41.398006 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 17024
W20260928 16:38:41.403832 17008 connection.cc:535] client connection to 127.0.0.1:39895 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39895): Connection reset by peer (error 104)
2026-09-28T16:38:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:41 dist-test-slave-g9sh krb5kdc[17125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:41 dist-test-slave-g9sh krb5kdc[17125](info): set up 2 sockets
Sep 28 16:38:41 dist-test-slave-g9sh krb5kdc[17125](info): commencing operation
krb5kdc: starting...
W20260928 16:38:42.435539 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:42 dist-test-slave-g9sh krb5kdc[17125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:38:42.554070 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:42.661854 17138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:42.662129 17138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:42.662206 17138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:42.666172 17138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:42.666249 17138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:42.666287 17138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:42.666347 17138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:42.666385 17138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:42.671450 17138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:42.672435 17138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:42.673497 17138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:42.681810 17144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:42.681814 17146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:42.681802 17143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:42.681991 17138 server_base.cc:1034] running on GCE node
I20260928 16:38:42.682571 17138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:42.683604 17138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:42.684795 17138 hybrid_clock.cc:648] HybridClock initialized: now 1790613522684764 us; error 41 us; skew 500 ppm
Sep 28 16:38:42 dist-test-slave-g9sh krb5kdc[17125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:42.687458 17138 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:38:42.688264 17138 webserver.cc:460] Webserver started at http://127.0.0.1:40679/ using document root <none> and password file <none>
I20260928 16:38:42.688810 17138 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:42.688903 17138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:42.689162 17138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:42.691164 17138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "a263a6f89d7f4f2d9da9a8eb52fe3b52"
format_stamp: "Formatted at 2026-09-28 16:38:42 on dist-test-slave-g9sh"
server_key: "0ae6a94d0fb3d46027417697e61b162c"
server_key_iv: "28dbfee16501356f744d7b1b03b89466"
server_key_version: "encryptionkey@0"
I20260928 16:38:42.691732 17138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "a263a6f89d7f4f2d9da9a8eb52fe3b52"
format_stamp: "Formatted at 2026-09-28 16:38:42 on dist-test-slave-g9sh"
server_key: "0ae6a94d0fb3d46027417697e61b162c"
server_key_iv: "28dbfee16501356f744d7b1b03b89466"
server_key_version: "encryptionkey@0"
I20260928 16:38:42.695307 17138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260928 16:38:42.697611 17153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:42.698825 17138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260928 16:38:42.698972 17138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "a263a6f89d7f4f2d9da9a8eb52fe3b52"
format_stamp: "Formatted at 2026-09-28 16:38:42 on dist-test-slave-g9sh"
server_key: "0ae6a94d0fb3d46027417697e61b162c"
server_key_iv: "28dbfee16501356f744d7b1b03b89466"
server_key_version: "encryptionkey@0"
I20260928 16:38:42.699082 17138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:42.703462 17138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:42.706265 17138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:42.706457 17138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:42.713050 17138 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43197
I20260928 16:38:42.713094 17207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43197 every 8 connection(s)
I20260928 16:38:42.713922 17138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:42.716472 17208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:42.719477 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 17138
I20260928 16:38:42.719628 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:42.719962 28139 external_mini_cluster.cc:1428] Setting key 20cc83672599fe4a0d6b5cbdcc313c06
I20260928 16:38:42.722352 17208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52: Bootstrap starting.
I20260928 16:38:42.724831 17208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:42.724864 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:42.721251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:34379 (local address 127.0.0.1:43197)
0928 16:38:42.721662 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:42.721677 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:42.721710 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:38:42.722303 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:42.722318 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:42.722635 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:42.722895 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:42.723690 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:42.723694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:42.723705 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:42.723715 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:42.723762 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:42.723780 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:42.723880 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:42.724000 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 16:38:42.724173 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":158,"threads_started":1}
I20260928 16:38:42.725744 17208 log.cc:824] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:42.727576 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:42.727793 17208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52: No bootstrap required, opened a new log
I20260928 16:38:42.731261 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:42.729623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:52545 (local address 127.0.0.1:43197)
0928 16:38:42.729738 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:42.729741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:42.729943 (+ 202us) server_negotiation.cc:407] Connection header received
0928 16:38:42.730096 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:42.730099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:42.730128 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:42.730220 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:42.730899 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:42.730903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:42.730905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:42.730916 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:42.730940 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:42.730951 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:42.731045 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:42.731117 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 16:38:42.731154 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:38:42.731540 17208 raft_consensus.cc:348] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a263a6f89d7f4f2d9da9a8eb52fe3b52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43197 } }
I20260928 16:38:42.731776 17208 raft_consensus.cc:374] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:42.731853 17208 raft_consensus.cc:729] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a263a6f89d7f4f2d9da9a8eb52fe3b52, State: Initialized, Role: FOLLOWER
I20260928 16:38:42.732304 17208 consensus_queue.cc:260] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a263a6f89d7f4f2d9da9a8eb52fe3b52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43197 } }
I20260928 16:38:42.732422 17208 raft_consensus.cc:388] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:42.732513 17208 raft_consensus.cc:482] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:42.732611 17208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:42.733515 17208 raft_consensus.cc:504] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a263a6f89d7f4f2d9da9a8eb52fe3b52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43197 } }
I20260928 16:38:42.733850 17208 leader_election.cc:302] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a263a6f89d7f4f2d9da9a8eb52fe3b52; no voters:
I20260928 16:38:42.734133 17208 leader_election.cc:288] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:42.734263 17218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:42.734503 17218 raft_consensus.cc:686] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [term 1 LEADER]: Becoming Leader. State: Replica: a263a6f89d7f4f2d9da9a8eb52fe3b52, State: Running, Role: LEADER
I20260928 16:38:42.734819 17218 consensus_queue.cc:237] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a263a6f89d7f4f2d9da9a8eb52fe3b52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43197 } }
I20260928 16:38:42.735285 17208 sys_catalog.cc:564] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:42.736435 17219 sys_catalog.cc:455] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a263a6f89d7f4f2d9da9a8eb52fe3b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a263a6f89d7f4f2d9da9a8eb52fe3b52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43197 } } }
I20260928 16:38:42.736442 17220 sys_catalog.cc:455] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a263a6f89d7f4f2d9da9a8eb52fe3b52. Latest consensus state: current_term: 1 leader_uuid: "a263a6f89d7f4f2d9da9a8eb52fe3b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a263a6f89d7f4f2d9da9a8eb52fe3b52" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43197 } } }
I20260928 16:38:42.736606 17219 sys_catalog.cc:458] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:42.736621 17220 sys_catalog.cc:458] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:42.737147 17227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:42.740229 17227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:42.745204 17227 catalog_manager.cc:1347] Generated new cluster ID: 4ad2473fec224b37a48743448fa6ab68
I20260928 16:38:42.745275 17227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:42.757436 17227 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:42.757730 17227 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:42.762403 17227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a263a6f89d7f4f2d9da9a8eb52fe3b52: Generated new TSK 0
I20260928 16:38:42.763001 17227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:42.766558 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 17138
2026-09-28T16:38:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:42 dist-test-slave-g9sh krb5kdc[17239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:42 dist-test-slave-g9sh krb5kdc[17239](info): set up 2 sockets
Sep 28 16:38:42 dist-test-slave-g9sh krb5kdc[17239](info): commencing operation
krb5kdc: starting...
W20260928 16:38:43.807451 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:43 dist-test-slave-g9sh krb5kdc[17239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:38:43.927973 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:44.033996 17252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:44.034296 17252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:44.034385 17252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:44.038385 17252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:44.038467 17252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:44.038506 17252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:44.038561 17252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:44.038599 17252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:44.043905 17252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37859
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:44.045007 17252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:44.046025 17252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:44.054363 17257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:44.054430 17260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:44.054363 17258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:44.054541 17252 server_base.cc:1034] running on GCE node
I20260928 16:38:44.055145 17252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:44.056115 17252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:44.057281 17252 hybrid_clock.cc:648] HybridClock initialized: now 1790613524057262 us; error 30 us; skew 500 ppm
Sep 28 16:38:44 dist-test-slave-g9sh krb5kdc[17239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:44.059931 17252 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:38:44.060737 17252 webserver.cc:460] Webserver started at http://127.0.0.1:39357/ using document root <none> and password file <none>
I20260928 16:38:44.061314 17252 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:44.061403 17252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:44.061628 17252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:44.063591 17252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "cbd2742b73e74461ba6bb7dcebe10bce"
format_stamp: "Formatted at 2026-09-28 16:38:44 on dist-test-slave-g9sh"
server_key: "bb918cbad0f74f546a3c603f81ec3b69"
server_key_iv: "94c5c1d438afb494e849918fe7a9faca"
server_key_version: "encryptionkey@0"
I20260928 16:38:44.064145 17252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "cbd2742b73e74461ba6bb7dcebe10bce"
format_stamp: "Formatted at 2026-09-28 16:38:44 on dist-test-slave-g9sh"
server_key: "bb918cbad0f74f546a3c603f81ec3b69"
server_key_iv: "94c5c1d438afb494e849918fe7a9faca"
server_key_version: "encryptionkey@0"
I20260928 16:38:44.067735 17252 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 16:38:44.070086 17267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:44.071270 17252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:38:44.071408 17252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "cbd2742b73e74461ba6bb7dcebe10bce"
format_stamp: "Formatted at 2026-09-28 16:38:44 on dist-test-slave-g9sh"
server_key: "bb918cbad0f74f546a3c603f81ec3b69"
server_key_iv: "94c5c1d438afb494e849918fe7a9faca"
server_key_version: "encryptionkey@0"
I20260928 16:38:44.071514 17252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:44.085000 17252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:44.088169 17252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:44.088363 17252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:44.095207 17252 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37859
I20260928 16:38:44.095232 17320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37859 every 8 connection(s)
I20260928 16:38:44.096082 17252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:44.098716 17321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:44.103140 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 17252
I20260928 16:38:44.103276 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:44.103580 28139 external_mini_cluster.cc:1428] Setting key 91bba690fadd657e40164a15abc61143
I20260928 16:38:44.104339 17321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce: Bootstrap starting.
I20260928 16:38:44.106590 17321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:44.107519 17321 log.cc:824] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:44.109546 17321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce: No bootstrap required, opened a new log
Sep 28 16:38:44 dist-test-slave-g9sh krb5kdc[17239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 16:38:44.113775 17321 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd2742b73e74461ba6bb7dcebe10bce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260928 16:38:44.114002 17321 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:44.114068 17321 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd2742b73e74461ba6bb7dcebe10bce, State: Initialized, Role: FOLLOWER
I20260928 16:38:44.114532 17321 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd2742b73e74461ba6bb7dcebe10bce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260928 16:38:44.114681 17321 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:44.114769 17321 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:44.114866 17321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:44.115660 17321 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd2742b73e74461ba6bb7dcebe10bce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260928 16:38:44.115984 17321 leader_election.cc:302] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbd2742b73e74461ba6bb7dcebe10bce; no voters:
I20260928 16:38:44.116300 17321 leader_election.cc:288] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:44.116357 17326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:44.116560 17326 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [term 1 LEADER]: Becoming Leader. State: Replica: cbd2742b73e74461ba6bb7dcebe10bce, State: Running, Role: LEADER
I20260928 16:38:44.117022 17326 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd2742b73e74461ba6bb7dcebe10bce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } }
I20260928 16:38:44.117568 17321 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:44.118467 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:44.104982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:58713 (local address 127.0.0.1:37859)
0928 16:38:44.105370 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:44.105391 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:44.105423 (+ 32us) server_negotiation.cc:407] Connection header received
0928 16:38:44.106015 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:44.106030 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:44.106351 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:44.106608 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:44.107530 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:44.108719 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:44.109860 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:44.110303 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:44.110424 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:44.113208 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:44.113213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:44.113225 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:44.113250 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:44.115102 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:44.115602 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:44.115608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:44.115616 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:44.115727 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:44.116094 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:44.116099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:44.116103 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:44.116598 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:44.117354 (+ 756us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:44.117481 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 16:38:44.117695 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":136,"threads_started":1}
I20260928 16:38:44.118842 17327 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbd2742b73e74461ba6bb7dcebe10bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd2742b73e74461ba6bb7dcebe10bce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } } }
I20260928 16:38:44.118959 17327 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:44.119339 17328 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbd2742b73e74461ba6bb7dcebe10bce. Latest consensus state: current_term: 1 leader_uuid: "cbd2742b73e74461ba6bb7dcebe10bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbd2742b73e74461ba6bb7dcebe10bce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37859 } } }
I20260928 16:38:44.119973 17328 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:44.120157 17335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:44.121687 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:44.123718 17335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:44.129739 17335 catalog_manager.cc:1347] Generated new cluster ID: bcf1f2c237e5491996b980c6f3b2b035
I20260928 16:38:44.129848 17335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:44.131532 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:44.123601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:36103 (local address 127.0.0.1:37859)
0928 16:38:44.123715 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:44.123719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:44.123945 (+ 226us) server_negotiation.cc:407] Connection header received
0928 16:38:44.124062 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:44.124065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:44.124106 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:44.124202 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:44.124830 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:44.125398 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:44.126199 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:44.126439 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:44.126547 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:44.128022 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:44.128026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:44.128028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:44.128051 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:44.129742 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:44.130271 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:44.130274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:44.130275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:44.130340 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:44.130673 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:44.130677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:44.130679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:44.130848 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:44.130997 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:44.131275 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 16:38:44.131366 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 16:38:44.161989 17335 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:44.162925 17335 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:44.168852 17335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbd2742b73e74461ba6bb7dcebe10bce: Generated new TSK 0
I20260928 16:38:44.169523 17335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:44.175865 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 17252
W20260928 16:38:44.181778 17237 connection.cc:535] client connection to 127.0.0.1:37859 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37859): Connection reset by peer (error 104)
2026-09-28T16:38:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:44 dist-test-slave-g9sh krb5kdc[17354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:44 dist-test-slave-g9sh krb5kdc[17354](info): set up 2 sockets
Sep 28 16:38:44 dist-test-slave-g9sh krb5kdc[17354](info): commencing operation
krb5kdc: starting...
W20260928 16:38:45.207430 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:45 dist-test-slave-g9sh krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:38:45.325572 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:45.433650 17367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:45.433876 17367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:45.433915 17367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:45.437850 17367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:45.437899 17367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:45.437918 17367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:45.437969 17367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:45.437996 17367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:45.443109 17367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:45.444031 17367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:45.445057 17367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:45.453228 17372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:45.453212 17375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:45.453223 17373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:45.453516 17367 server_base.cc:1034] running on GCE node
I20260928 16:38:45.454051 17367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:45.455065 17367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:45.456259 17367 hybrid_clock.cc:648] HybridClock initialized: now 1790613525456238 us; error 29 us; skew 500 ppm
Sep 28 16:38:45 dist-test-slave-g9sh krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:45.464800 17367 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:38:45.465708 17367 webserver.cc:460] Webserver started at http://127.0.0.1:32961/ using document root <none> and password file <none>
I20260928 16:38:45.466205 17367 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:45.466262 17367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:45.466418 17367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:45.468333 17367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "3c255f0430844802bf092b4378bb4a92"
format_stamp: "Formatted at 2026-09-28 16:38:45 on dist-test-slave-g9sh"
server_key: "1fecb96081f2f2e57f4a5992501e85d4"
server_key_iv: "adac4ac5e41e71315c972bd625aada88"
server_key_version: "encryptionkey@0"
I20260928 16:38:45.468835 17367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "3c255f0430844802bf092b4378bb4a92"
format_stamp: "Formatted at 2026-09-28 16:38:45 on dist-test-slave-g9sh"
server_key: "1fecb96081f2f2e57f4a5992501e85d4"
server_key_iv: "adac4ac5e41e71315c972bd625aada88"
server_key_version: "encryptionkey@0"
I20260928 16:38:45.472389 17367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:38:45.474692 17383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:45.475876 17367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:38:45.476006 17367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "3c255f0430844802bf092b4378bb4a92"
format_stamp: "Formatted at 2026-09-28 16:38:45 on dist-test-slave-g9sh"
server_key: "1fecb96081f2f2e57f4a5992501e85d4"
server_key_iv: "adac4ac5e41e71315c972bd625aada88"
server_key_version: "encryptionkey@0"
I20260928 16:38:45.476105 17367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:45.489235 17367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:45.492043 17367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:45.492240 17367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:45.499359 17367 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37539
I20260928 16:38:45.499380 17436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37539 every 8 connection(s)
I20260928 16:38:45.500262 17367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:45.500942 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 17367
I20260928 16:38:45.501114 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:45.501403 28139 external_mini_cluster.cc:1428] Setting key 35c6934aabd8d8cf556073b87a34affe
I20260928 16:38:45.503535 17437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:45.510265 17437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92: Bootstrap starting.
I20260928 16:38:45.511323 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:45.502810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:50261 (local address 127.0.0.1:37539)
0928 16:38:45.503224 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:45.503243 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:45.503285 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:38:45.504109 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:45.504131 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:45.504539 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:45.504890 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:45.505320 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:45.506798 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:45.507677 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:45.508407 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:45.508564 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:45.509445 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:45.509452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:45.509466 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:45.509954 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:45.510020 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:45.510047 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:45.510179 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:45.510349 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 16:38:45.510477 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":110,"threads_started":1}
I20260928 16:38:45.512624 17437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:45.513505 17437 log.cc:824] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:45.514161 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:45.515415 17437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92: No bootstrap required, opened a new log
I20260928 16:38:45.519518 17437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c255f0430844802bf092b4378bb4a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260928 16:38:45.519780 17437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:45.519853 17437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c255f0430844802bf092b4378bb4a92, State: Initialized, Role: FOLLOWER
I20260928 16:38:45.520213 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:45.516109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:53793 (local address 127.0.0.1:37539)
0928 16:38:45.516231 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:45.516236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:45.516453 (+ 217us) server_negotiation.cc:407] Connection header received
0928 16:38:45.516536 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:45.516539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:45.516591 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:45.516664 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:45.516885 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:45.517724 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:45.518508 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:45.518849 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:45.518958 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:45.519708 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:45.519712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:45.519714 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:45.519726 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:45.519754 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:45.519767 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:45.519864 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:45.520059 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 16:38:45.520096 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 16:38:45.520376 17437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c255f0430844802bf092b4378bb4a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260928 16:38:45.520502 17437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:45.520550 17437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:45.520665 17437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:45.521556 17437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c255f0430844802bf092b4378bb4a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260928 16:38:45.521900 17437 leader_election.cc:302] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c255f0430844802bf092b4378bb4a92; no voters:
I20260928 16:38:45.522187 17437 leader_election.cc:288] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:45.522330 17447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:45.522557 17447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [term 1 LEADER]: Becoming Leader. State: Replica: 3c255f0430844802bf092b4378bb4a92, State: Running, Role: LEADER
I20260928 16:38:45.522893 17447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c255f0430844802bf092b4378bb4a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } }
I20260928 16:38:45.523352 17437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:45.524542 17450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c255f0430844802bf092b4378bb4a92. Latest consensus state: current_term: 1 leader_uuid: "3c255f0430844802bf092b4378bb4a92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c255f0430844802bf092b4378bb4a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } } }
I20260928 16:38:45.524670 17450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:45.525139 17448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c255f0430844802bf092b4378bb4a92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c255f0430844802bf092b4378bb4a92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37539 } } }
I20260928 16:38:45.525238 17448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:45.525198 17457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:45.527782 17457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:45.533145 17457 catalog_manager.cc:1347] Generated new cluster ID: 618ace68ac1d4e1c9150d52c3a329159
I20260928 16:38:45.533217 17457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:45.542208 17457 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:45.543032 17457 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:45.548038 17457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c255f0430844802bf092b4378bb4a92: Generated new TSK 0
I20260928 16:38:45.548786 17457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:45.558146 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 17367
W20260928 16:38:45.563845 17352 connection.cc:535] client connection to 127.0.0.1:37539 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37539): Connection reset by peer (error 104)
2026-09-28T16:38:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:45 dist-test-slave-g9sh krb5kdc[17469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:45 dist-test-slave-g9sh krb5kdc[17469](info): set up 2 sockets
Sep 28 16:38:45 dist-test-slave-g9sh krb5kdc[17469](info): commencing operation
krb5kdc: starting...
W20260928 16:38:46.603467 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:46 dist-test-slave-g9sh krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 16:38:46.724035 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:46.829929 17482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:46.830201 17482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:46.830276 17482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:46.834198 17482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:46.834290 17482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:46.834327 17482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:46.834360 17482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:46.834389 17482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:46.839437 17482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41073
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:46.840392 17482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:46.841377 17482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:46.849454 17487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:46.849506 17488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:46.849515 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:46.849772 17482 server_base.cc:1034] running on GCE node
I20260928 16:38:46.850314 17482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:46.851322 17482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:46.852488 17482 hybrid_clock.cc:648] HybridClock initialized: now 1790613526852470 us; error 28 us; skew 500 ppm
Sep 28 16:38:46 dist-test-slave-g9sh krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:46.855347 17482 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 16:38:46.856168 17482 webserver.cc:460] Webserver started at http://127.0.0.1:35615/ using document root <none> and password file <none>
I20260928 16:38:46.856699 17482 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:46.856791 17482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:46.857069 17482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:46.859015 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "771be003a6b24872b9424f31261aa119"
format_stamp: "Formatted at 2026-09-28 16:38:46 on dist-test-slave-g9sh"
server_key: "482378c2c3431333afc4666cd908a9ee"
server_key_iv: "e0833bda683780456bb0cb75b901beef"
server_key_version: "encryptionkey@0"
I20260928 16:38:46.859570 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "771be003a6b24872b9424f31261aa119"
format_stamp: "Formatted at 2026-09-28 16:38:46 on dist-test-slave-g9sh"
server_key: "482378c2c3431333afc4666cd908a9ee"
server_key_iv: "e0833bda683780456bb0cb75b901beef"
server_key_version: "encryptionkey@0"
I20260928 16:38:46.863117 17482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:38:46.865381 17497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:46.866609 17482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:46.866747 17482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "771be003a6b24872b9424f31261aa119"
format_stamp: "Formatted at 2026-09-28 16:38:46 on dist-test-slave-g9sh"
server_key: "482378c2c3431333afc4666cd908a9ee"
server_key_iv: "e0833bda683780456bb0cb75b901beef"
server_key_version: "encryptionkey@0"
I20260928 16:38:46.866854 17482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:46.871264 17482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:46.874366 17482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:46.874562 17482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:46.881366 17482 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41073
I20260928 16:38:46.881378 17550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41073 every 8 connection(s)
I20260928 16:38:46.882229 17482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:46.884867 17551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:46.889174 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 17482
I20260928 16:38:46.889333 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:46.889623 28139 external_mini_cluster.cc:1428] Setting key 620952e8e969391985ee4c46f32283c4
I20260928 16:38:46.890342 17551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119: Bootstrap starting.
I20260928 16:38:46.892627 17551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:46.893590 17551 log.cc:824] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:46.894441 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:46.890887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:50381 (local address 127.0.0.1:41073)
0928 16:38:46.891292 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:46.891306 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:46.891340 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:38:46.891949 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:46.891965 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:46.892231 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:46.892496 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:46.893316 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:46.893320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:46.893331 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:46.893341 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:46.893390 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:46.893408 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:46.893496 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:46.893640 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 16:38:46.893805 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":128,"threads_started":1}
I20260928 16:38:46.895511 17551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119: No bootstrap required, opened a new log
I20260928 16:38:46.897282 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:46.899572 17551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771be003a6b24872b9424f31261aa119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } }
I20260928 16:38:46.899806 17551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:46.899875 17551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771be003a6b24872b9424f31261aa119, State: Initialized, Role: FOLLOWER
I20260928 16:38:46.900362 17551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771be003a6b24872b9424f31261aa119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } }
I20260928 16:38:46.900523 17551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:46.900592 17551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:46.900703 17551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:46.900983 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:46.899255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.252:56357 (local address 127.0.0.1:41073)
0928 16:38:46.899367 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:46.899371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:46.899540 (+ 169us) server_negotiation.cc:407] Connection header received
0928 16:38:46.899675 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:46.899678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:46.899710 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:46.899791 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:46.900484 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:46.900487 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:46.900490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:46.900522 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:46.900550 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:46.900563 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:46.900643 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:46.900766 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 16:38:46.900823 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 16:38:46.901853 17551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771be003a6b24872b9424f31261aa119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } }
I20260928 16:38:46.902226 17551 leader_election.cc:302] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 771be003a6b24872b9424f31261aa119; no voters:
I20260928 16:38:46.902590 17551 leader_election.cc:288] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:46.902678 17561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:46.902937 17561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [term 1 LEADER]: Becoming Leader. State: Replica: 771be003a6b24872b9424f31261aa119, State: Running, Role: LEADER
I20260928 16:38:46.903286 17561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771be003a6b24872b9424f31261aa119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } }
I20260928 16:38:46.903723 17551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:46.904816 17563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 771be003a6b24872b9424f31261aa119. Latest consensus state: current_term: 1 leader_uuid: "771be003a6b24872b9424f31261aa119" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771be003a6b24872b9424f31261aa119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } } }
I20260928 16:38:46.904965 17563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:46.905282 17562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "771be003a6b24872b9424f31261aa119" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771be003a6b24872b9424f31261aa119" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41073 } } }
I20260928 16:38:46.905373 17571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:46.905426 17562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:46.907999 17571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:46.913221 17571 catalog_manager.cc:1347] Generated new cluster ID: bcff056bc29343c69da5e5a5c556d652
I20260928 16:38:46.913285 17571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:46.930203 17571 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:46.930517 17571 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:46.934532 17571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 771be003a6b24872b9424f31261aa119: Generated new TSK 0
I20260928 16:38:46.935125 17571 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:46.937690 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 17482
2026-09-28T16:38:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:46 dist-test-slave-g9sh krb5kdc[17583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:46 dist-test-slave-g9sh krb5kdc[17583](info): set up 2 sockets
Sep 28 16:38:46 dist-test-slave-g9sh krb5kdc[17583](info): commencing operation
krb5kdc: starting...
W20260928 16:38:47.979933 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:48 dist-test-slave-g9sh krb5kdc[17583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:38:48.096498 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:40935
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:40935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:48.204282 17596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:48.204566 17596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:48.204649 17596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:48.208619 17596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:48.208709 17596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:48.208750 17596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:48.208806 17596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:48.208846 17596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:48.214576 17596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:40935
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:48.215592 17596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:48.216586 17596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:38:48.224951 17596 server_base.cc:1034] running on GCE node
W20260928 16:38:48.224884 17602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:48.224921 17601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:48.224854 17604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:48.225721 17596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:48.226747 17596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:48.227932 17596 hybrid_clock.cc:648] HybridClock initialized: now 1790613528227906 us; error 45 us; skew 500 ppm
Sep 28 16:38:48 dist-test-slave-g9sh krb5kdc[17583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613528, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:48.230559 17596 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:38:48.231458 17596 webserver.cc:460] Webserver started at http://127.27.122.254:41773/ using document root <none> and password file <none>
I20260928 16:38:48.232008 17596 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:48.232096 17596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:48.232316 17596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:48.234339 17596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "14120f2bff06431ab818ceb4be7f1112"
format_stamp: "Formatted at 2026-09-28 16:38:48 on dist-test-slave-g9sh"
server_key: "e51c7207c135442a57c05b213a96f7ba"
server_key_iv: "b76db62e0e998720e76e58ed6e3cba5d"
server_key_version: "encryptionkey@0"
I20260928 16:38:48.234894 17596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "14120f2bff06431ab818ceb4be7f1112"
format_stamp: "Formatted at 2026-09-28 16:38:48 on dist-test-slave-g9sh"
server_key: "e51c7207c135442a57c05b213a96f7ba"
server_key_iv: "b76db62e0e998720e76e58ed6e3cba5d"
server_key_version: "encryptionkey@0"
I20260928 16:38:48.238417 17596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:38:48.240674 17611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:48.241876 17596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:38:48.242019 17596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "14120f2bff06431ab818ceb4be7f1112"
format_stamp: "Formatted at 2026-09-28 16:38:48 on dist-test-slave-g9sh"
server_key: "e51c7207c135442a57c05b213a96f7ba"
server_key_iv: "b76db62e0e998720e76e58ed6e3cba5d"
server_key_version: "encryptionkey@0"
I20260928 16:38:48.242126 17596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:48.256072 17596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:48.258952 17596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:48.259146 17596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:48.266317 17596 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:40935
I20260928 16:38:48.266340 17664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:40935 every 8 connection(s)
I20260928 16:38:48.267207 17596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:48.269793 17665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:48.271919 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 17596
I20260928 16:38:48.272060 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:48.272328 28139 external_mini_cluster.cc:1428] Setting key cf36582deb1f6e007dea710b10bcdd90
I20260928 16:38:48.276098 17665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112: Bootstrap starting.
I20260928 16:38:48.278702 17665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:48.279587 17665 log.cc:824] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:48.281533 17665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112: No bootstrap required, opened a new log
Sep 28 16:38:48 dist-test-slave-g9sh krb5kdc[17583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:48.286185 17665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14120f2bff06431ab818ceb4be7f1112" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40935 } }
I20260928 16:38:48.286460 17665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:48.286576 17665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14120f2bff06431ab818ceb4be7f1112, State: Initialized, Role: FOLLOWER
I20260928 16:38:48.287045 17665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14120f2bff06431ab818ceb4be7f1112" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40935 } }
I20260928 16:38:48.287221 17665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:48.287303 17665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:48.287395 17665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:48.287843 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:48.273725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33285 (local address 127.27.122.254:40935)
0928 16:38:48.274148 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:48.274173 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:48.274219 (+ 46us) server_negotiation.cc:407] Connection header received
0928 16:38:48.275039 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:48.275063 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:48.275479 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:48.275836 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:48.276642 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:48.278156 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:48.279057 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:48.279777 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:48.279953 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:48.282537 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:48.282543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:48.282558 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:48.282589 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:48.284614 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:48.285185 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:48.285191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:48.285196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:48.285337 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:48.285671 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:48.285674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:48.285676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:48.286146 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:48.286373 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:48.286603 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 16:38:48.286837 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":96,"threads_started":1}
I20260928 16:38:48.288446 17665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14120f2bff06431ab818ceb4be7f1112" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40935 } }
I20260928 16:38:48.288784 17665 leader_election.cc:302] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14120f2bff06431ab818ceb4be7f1112; no voters:
I20260928 16:38:48.289121 17665 leader_election.cc:288] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:48.289222 17670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:48.289454 17670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [term 1 LEADER]: Becoming Leader. State: Replica: 14120f2bff06431ab818ceb4be7f1112, State: Running, Role: LEADER
I20260928 16:38:48.289824 17670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14120f2bff06431ab818ceb4be7f1112" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40935 } }
I20260928 16:38:48.290634 17665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:48.291397 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:48.291450 17671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14120f2bff06431ab818ceb4be7f1112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14120f2bff06431ab818ceb4be7f1112" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40935 } } }
I20260928 16:38:48.291553 17671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:48.291841 17672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14120f2bff06431ab818ceb4be7f1112. Latest consensus state: current_term: 1 leader_uuid: "14120f2bff06431ab818ceb4be7f1112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14120f2bff06431ab818ceb4be7f1112" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 40935 } } }
I20260928 16:38:48.291929 17672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112 [sys.catalog]: This master's current role is: LEADER
W20260928 16:38:48.295950 17689 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:38:48.296072 17689 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:38:48.296144 17679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:48.297008 17679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:48.301270 17668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:48.293661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57065 (local address 127.27.122.254:40935)
0928 16:38:48.293791 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:48.293795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:48.293933 (+ 138us) server_negotiation.cc:407] Connection header received
0928 16:38:48.294020 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:48.294023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:48.294065 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:48.294138 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:48.294742 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:48.295239 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:48.296039 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:48.296339 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:48.296449 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:48.297815 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:48.297819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:48.297821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:48.297846 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:48.299280 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:48.299885 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:48.299888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:48.299890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:48.299974 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:48.300339 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:48.300342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:48.300343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:48.300545 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:48.300695 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:48.301033 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 16:38:48.301127 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:38:48.303179 17679 catalog_manager.cc:1347] Generated new cluster ID: 14d27392ff824ad09bf891000d5b592e
I20260928 16:38:48.303236 17679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:48.320225 17679 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:48.321238 17679 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:48.327486 17679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14120f2bff06431ab818ceb4be7f1112: Generated new TSK 0
I20260928 16:38:48.328205 17679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:48.340396 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 17596
W20260928 16:38:48.346505 17581 connection.cc:535] client connection to 127.27.122.254:40935 recv error: Network error: failed to read from TLS socket (remote: 127.27.122.254:40935): Connection reset by peer (error 104)
2026-09-28T16:38:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:48 dist-test-slave-g9sh krb5kdc[17696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:48 dist-test-slave-g9sh krb5kdc[17696](info): set up 2 sockets
Sep 28 16:38:48 dist-test-slave-g9sh krb5kdc[17696](info): commencing operation
krb5kdc: starting...
W20260928 16:38:49.371582 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:49 dist-test-slave-g9sh krb5kdc[17696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:38:49.491122 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:39925
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:42723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:39925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:49.612448 17709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:49.612672 17709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:49.612711 17709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:49.616706 17709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:49.616760 17709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:49.616779 17709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:49.616796 17709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:49.616811 17709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:49.621793 17709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:42723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:39925
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:39925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:49.622658 17709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:49.623538 17709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:49.631880 17715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:49.631880 17714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:49.631897 17717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:49.632297 17709 server_base.cc:1034] running on GCE node
I20260928 16:38:49.632820 17709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:49.633855 17709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:49.635012 17709 hybrid_clock.cc:648] HybridClock initialized: now 1790613529634984 us; error 41 us; skew 500 ppm
Sep 28 16:38:49 dist-test-slave-g9sh krb5kdc[17696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613529, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:49.637856 17709 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:38:49.638780 17709 webserver.cc:460] Webserver started at http://127.27.122.254:36645/ using document root <none> and password file <none>
I20260928 16:38:49.639355 17709 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:49.639417 17709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:49.639636 17709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:49.641703 17709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "3e3eb9e01eda4b0e945fa6659b8433d5"
format_stamp: "Formatted at 2026-09-28 16:38:49 on dist-test-slave-g9sh"
server_key: "1322879285dda2d97157353536293294"
server_key_iv: "63534ac2d27a626ac9b0aeb2dc2a9d43"
server_key_version: "encryptionkey@0"
I20260928 16:38:49.642258 17709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "3e3eb9e01eda4b0e945fa6659b8433d5"
format_stamp: "Formatted at 2026-09-28 16:38:49 on dist-test-slave-g9sh"
server_key: "1322879285dda2d97157353536293294"
server_key_iv: "63534ac2d27a626ac9b0aeb2dc2a9d43"
server_key_version: "encryptionkey@0"
I20260928 16:38:49.646076 17709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 16:38:49.648402 17724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:49.649569 17709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:38:49.649712 17709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "3e3eb9e01eda4b0e945fa6659b8433d5"
format_stamp: "Formatted at 2026-09-28 16:38:49 on dist-test-slave-g9sh"
server_key: "1322879285dda2d97157353536293294"
server_key_iv: "63534ac2d27a626ac9b0aeb2dc2a9d43"
server_key_version: "encryptionkey@0"
I20260928 16:38:49.649822 17709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:49.670156 17709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:49.673338 17709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:49.673539 17709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:49.680693 17709 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:39925
I20260928 16:38:49.680722 17776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:39925 every 8 connection(s)
I20260928 16:38:49.681674 17709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:49.684305 17777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:49.686576 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 17709
I20260928 16:38:49.686733 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:49.687016 28139 external_mini_cluster.cc:1428] Setting key 3908adb8aff788f35b7d1f1f1c0318be
I20260928 16:38:49.690531 17777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5: Bootstrap starting.
I20260928 16:38:49.693118 17777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:49.694015 17777 log.cc:824] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:49.696192 17777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5: No bootstrap required, opened a new log
I20260928 16:38:49.697178 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:49.688396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52545 (local address 127.27.122.254:39925)
0928 16:38:49.688818 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:49.688836 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:49.688878 (+ 42us) server_negotiation.cc:407] Connection header received
0928 16:38:49.689741 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:49.689765 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:49.690177 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:49.690517 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:49.690914 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:49.692376 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:49.693446 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:49.694138 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:49.694296 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:49.695251 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:49.695257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:49.695271 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:49.695733 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:49.695798 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:49.695827 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:49.695957 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:49.696132 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 16:38:49.696259 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":122,"threads_started":1}
I20260928 16:38:49.700031 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:49.700325 17777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3eb9e01eda4b0e945fa6659b8433d5" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 39925 } }
I20260928 16:38:49.700547 17777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:49.700629 17777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e3eb9e01eda4b0e945fa6659b8433d5, State: Initialized, Role: FOLLOWER
I20260928 16:38:49.701092 17777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3eb9e01eda4b0e945fa6659b8433d5" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 39925 } }
I20260928 16:38:49.701237 17777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:49.701308 17777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:49.701426 17777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:49.702432 17777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3eb9e01eda4b0e945fa6659b8433d5" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 39925 } }
I20260928 16:38:49.702816 17777 leader_election.cc:302] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e3eb9e01eda4b0e945fa6659b8433d5; no voters:
I20260928 16:38:49.703173 17777 leader_election.cc:288] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:49.703235 17787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:49.703437 17787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3e3eb9e01eda4b0e945fa6659b8433d5, State: Running, Role: LEADER
I20260928 16:38:49.703823 17787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3eb9e01eda4b0e945fa6659b8433d5" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 39925 } }
I20260928 16:38:49.704535 17777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:49.705181 17789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e3eb9e01eda4b0e945fa6659b8433d5. Latest consensus state: current_term: 1 leader_uuid: "3e3eb9e01eda4b0e945fa6659b8433d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3eb9e01eda4b0e945fa6659b8433d5" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 39925 } } }
I20260928 16:38:49.706257 17789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:49.706143 17788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e3eb9e01eda4b0e945fa6659b8433d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3eb9e01eda4b0e945fa6659b8433d5" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 39925 } } }
I20260928 16:38:49.706594 17788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:49.706674 17800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:49.706962 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:49.702081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33231 (local address 127.27.122.254:39925)
0928 16:38:49.702198 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:49.702202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:49.702482 (+ 280us) server_negotiation.cc:407] Connection header received
0928 16:38:49.702585 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:49.702588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:49.702645 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:49.702714 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:49.703041 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:49.703762 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:49.704760 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:49.705135 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:49.705266 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:49.706172 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:49.706176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:49.706179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:49.706191 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:49.706217 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:49.706233 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:49.706335 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:49.706813 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 16:38:49.706846 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:38:49.709908 17800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:49.714980 17800 catalog_manager.cc:1347] Generated new cluster ID: 377d449c2406400ea0338498666f3dc1
I20260928 16:38:49.715075 17800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:49.724097 17800 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:49.725265 17800 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:49.729460 17800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e3eb9e01eda4b0e945fa6659b8433d5: Generated new TSK 0
I20260928 16:38:49.730141 17800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:49.735662 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 17709
W20260928 16:38:49.741325 17694 connection.cc:535] client connection to 127.27.122.254:39925 recv error: Network error: failed to read from TLS socket (remote: 127.27.122.254:39925): Connection reset by peer (error 104)
2026-09-28T16:38:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:49 dist-test-slave-g9sh krb5kdc[17809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:49 dist-test-slave-g9sh krb5kdc[17809](info): set up 2 sockets
Sep 28 16:38:49 dist-test-slave-g9sh krb5kdc[17809](info): commencing operation
krb5kdc: starting...
W20260928 16:38:50.763561 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:50 dist-test-slave-g9sh krb5kdc[17809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:38:50.883599 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:46687
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:37707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:46687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:50.993451 17822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:50.993685 17822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:50.993726 17822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:50.997705 17822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:50.997759 17822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:50.997779 17822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:50.997795 17822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:50.997810 17822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:51.002931 17822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:37707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:46687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:46687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:51.003885 17822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:51.004864 17822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:38:51.013156 17822 server_base.cc:1034] running on GCE node
W20260928 16:38:51.013113 17829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:51.013123 17828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:51.013137 17831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:51.013998 17822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:51.015005 17822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:51.016197 17822 hybrid_clock.cc:648] HybridClock initialized: now 1790613531016190 us; error 28 us; skew 500 ppm
Sep 28 16:38:51 dist-test-slave-g9sh krb5kdc[17809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613531, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:51.024051 17822 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:38:51.025005 17822 webserver.cc:460] Webserver started at http://127.27.122.254:37201/ using document root <none> and password file <none>
I20260928 16:38:51.025544 17822 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:51.025635 17822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:51.025861 17822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:51.027807 17822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "90fb41afa5c741eb8cc6a5e0e47bbfbf"
format_stamp: "Formatted at 2026-09-28 16:38:51 on dist-test-slave-g9sh"
server_key: "d40714350b70bf2df9a0558f9a3ffe3b"
server_key_iv: "c951c0fb0d0adc0b9c725a7168bc8cfb"
server_key_version: "encryptionkey@0"
I20260928 16:38:51.028376 17822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "90fb41afa5c741eb8cc6a5e0e47bbfbf"
format_stamp: "Formatted at 2026-09-28 16:38:51 on dist-test-slave-g9sh"
server_key: "d40714350b70bf2df9a0558f9a3ffe3b"
server_key_iv: "c951c0fb0d0adc0b9c725a7168bc8cfb"
server_key_version: "encryptionkey@0"
I20260928 16:38:51.031949 17822 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:38:51.034288 17838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:51.035427 17822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:38:51.035568 17822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "90fb41afa5c741eb8cc6a5e0e47bbfbf"
format_stamp: "Formatted at 2026-09-28 16:38:51 on dist-test-slave-g9sh"
server_key: "d40714350b70bf2df9a0558f9a3ffe3b"
server_key_iv: "c951c0fb0d0adc0b9c725a7168bc8cfb"
server_key_version: "encryptionkey@0"
I20260928 16:38:51.035673 17822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:51.040135 17822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:51.042908 17822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:51.043129 17822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:51.050086 17822 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:46687
I20260928 16:38:51.050112 17890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:46687 every 8 connection(s)
I20260928 16:38:51.051133 17822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:51.053826 17891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:51.058915 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 17822
I20260928 16:38:51.059075 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:51.059398 28139 external_mini_cluster.cc:1428] Setting key fe2d3e1f215a9507d38a7fa5b015d411
I20260928 16:38:51.059473 17891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf: Bootstrap starting.
I20260928 16:38:51.061864 17891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:51.062582 17891 log.cc:824] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:51.064332 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:51.060724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43377 (local address 127.27.122.254:46687)
0928 16:38:51.061146 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:51.061159 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:51.061193 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:38:51.061783 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:51.061799 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:51.062116 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:51.062439 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:51.063227 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:51.063239 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:51.063250 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:51.063260 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:51.063319 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:51.063339 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:51.063450 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:51.063555 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 16:38:51.063707 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":140,"threads_started":1}
I20260928 16:38:51.064471 17891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf: No bootstrap required, opened a new log
I20260928 16:38:51.067184 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:51.068506 17891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fb41afa5c741eb8cc6a5e0e47bbfbf" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46687 } }
I20260928 16:38:51.068707 17891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:51.068778 17891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90fb41afa5c741eb8cc6a5e0e47bbfbf, State: Initialized, Role: FOLLOWER
I20260928 16:38:51.069278 17891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fb41afa5c741eb8cc6a5e0e47bbfbf" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46687 } }
I20260928 16:38:51.069453 17891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:51.069525 17891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:51.069622 17891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:51.070659 17894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:51.069141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59077 (local address 127.27.122.254:46687)
0928 16:38:51.069247 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:51.069253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:51.069430 (+ 177us) server_negotiation.cc:407] Connection header received
0928 16:38:51.069551 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:51.069554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:51.069582 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:51.069652 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:51.070317 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:51.070320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:51.070322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:51.070332 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:51.070354 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:51.070364 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:51.070428 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:51.070549 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 16:38:51.070578 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 16:38:51.070770 17891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fb41afa5c741eb8cc6a5e0e47bbfbf" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46687 } }
I20260928 16:38:51.071151 17891 leader_election.cc:302] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90fb41afa5c741eb8cc6a5e0e47bbfbf; no voters:
I20260928 16:38:51.071460 17891 leader_election.cc:288] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:51.071576 17901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:51.071817 17901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [term 1 LEADER]: Becoming Leader. State: Replica: 90fb41afa5c741eb8cc6a5e0e47bbfbf, State: Running, Role: LEADER
I20260928 16:38:51.072144 17901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fb41afa5c741eb8cc6a5e0e47bbfbf" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46687 } }
I20260928 16:38:51.072643 17891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:51.073765 17903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90fb41afa5c741eb8cc6a5e0e47bbfbf. Latest consensus state: current_term: 1 leader_uuid: "90fb41afa5c741eb8cc6a5e0e47bbfbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fb41afa5c741eb8cc6a5e0e47bbfbf" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46687 } } }
I20260928 16:38:51.073909 17903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:51.074447 17902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90fb41afa5c741eb8cc6a5e0e47bbfbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90fb41afa5c741eb8cc6a5e0e47bbfbf" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46687 } } }
I20260928 16:38:51.074563 17902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf [sys.catalog]: This master's current role is: LEADER
W20260928 16:38:51.077390 17916 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:38:51.077453 17916 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:38:51.077543 17910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:51.078231 17910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:51.083374 17910 catalog_manager.cc:1347] Generated new cluster ID: ec518a046a4948a397bc02d6c5435d0f
I20260928 16:38:51.083431 17910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:51.096433 17910 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:51.096753 17910 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:51.108459 17910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90fb41afa5c741eb8cc6a5e0e47bbfbf: Generated new TSK 0
I20260928 16:38:51.109254 17910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:51.112856 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 17822
2026-09-28T16:38:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:51 dist-test-slave-g9sh krb5kdc[17923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:51 dist-test-slave-g9sh krb5kdc[17923](info): set up 2 sockets
Sep 28 16:38:51 dist-test-slave-g9sh krb5kdc[17923](info): commencing operation
krb5kdc: starting...
W20260928 16:38:52.155792 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:52 dist-test-slave-g9sh krb5kdc[17923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:38:52.276407 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:38955
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:38955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:52.383150 17936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:52.383399 17936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:52.383440 17936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:52.387485 17936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:52.387540 17936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:52.387596 17936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:52.387635 17936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:52.387660 17936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:52.392802 17936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:38955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:38955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:52.393834 17936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:52.394835 17936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:38:52.403226 17936 server_base.cc:1034] running on GCE node
W20260928 16:38:52.403191 17942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:52.403191 17941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:52.403198 17944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:52.404089 17936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:52.405189 17936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:52.406374 17936 hybrid_clock.cc:648] HybridClock initialized: now 1790613532406355 us; error 29 us; skew 500 ppm
Sep 28 16:38:52 dist-test-slave-g9sh krb5kdc[17923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613532, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:52.409030 17936 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:38:52.409839 17936 webserver.cc:460] Webserver started at http://127.27.122.254:35205/ using document root <none> and password file <none>
I20260928 16:38:52.410368 17936 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:52.410461 17936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:52.410698 17936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:52.412683 17936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "e2423c036d644a9b9133288fc1b641d8"
format_stamp: "Formatted at 2026-09-28 16:38:52 on dist-test-slave-g9sh"
server_key: "4c30533d47dfce90b2053ef2ea46d097"
server_key_iv: "ef5896fb3ca994a2b27b2cc034c17315"
server_key_version: "encryptionkey@0"
I20260928 16:38:52.413286 17936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "e2423c036d644a9b9133288fc1b641d8"
format_stamp: "Formatted at 2026-09-28 16:38:52 on dist-test-slave-g9sh"
server_key: "4c30533d47dfce90b2053ef2ea46d097"
server_key_iv: "ef5896fb3ca994a2b27b2cc034c17315"
server_key_version: "encryptionkey@0"
I20260928 16:38:52.416826 17936 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 16:38:52.419139 17952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:52.420398 17936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 16:38:52.420542 17936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "e2423c036d644a9b9133288fc1b641d8"
format_stamp: "Formatted at 2026-09-28 16:38:52 on dist-test-slave-g9sh"
server_key: "4c30533d47dfce90b2053ef2ea46d097"
server_key_iv: "ef5896fb3ca994a2b27b2cc034c17315"
server_key_version: "encryptionkey@0"
I20260928 16:38:52.420657 17936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:52.435945 17936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:52.438819 17936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:52.439018 17936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:52.446337 17936 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:38955
I20260928 16:38:52.446365 18004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:38955 every 8 connection(s)
I20260928 16:38:52.447248 17936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:52.449982 18005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:52.451915 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 17936
I20260928 16:38:52.452066 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:52.452344 28139 external_mini_cluster.cc:1428] Setting key 661a79176df5e4ba982f14d8c06cfabd
I20260928 16:38:52.457027 18005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8: Bootstrap starting.
I20260928 16:38:52.459781 18005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:52.460738 18005 log.cc:824] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8: Log is configured to *not* fsync() on all Append() calls
Sep 28 16:38:52 dist-test-slave-g9sh krb5kdc[17923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:52.463073 18005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8: No bootstrap required, opened a new log
I20260928 16:38:52.468207 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:52.453767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41221 (local address 127.27.122.254:38955)
0928 16:38:52.454192 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:52.454217 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:52.454261 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:38:52.455131 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:52.455156 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:52.455608 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:52.455973 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:52.456712 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:52.458192 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:52.459087 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:52.459834 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:52.460037 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:52.462934 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:52.462941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:52.462956 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:52.462988 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:52.465009 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:52.465446 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:52.465453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:52.465462 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:52.465658 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:52.465938 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:52.465942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:52.465944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:52.466496 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:52.466708 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:52.466928 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 16:38:52.467234 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":105,"threads_started":1}
I20260928 16:38:52.468055 18005 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2423c036d644a9b9133288fc1b641d8" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38955 } }
I20260928 16:38:52.468308 18005 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:52.468366 18005 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2423c036d644a9b9133288fc1b641d8, State: Initialized, Role: FOLLOWER
I20260928 16:38:52.468880 18005 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2423c036d644a9b9133288fc1b641d8" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38955 } }
I20260928 16:38:52.469100 18005 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:52.469172 18005 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:52.469267 18005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:52.470163 18005 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2423c036d644a9b9133288fc1b641d8" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38955 } }
I20260928 16:38:52.470537 18005 leader_election.cc:302] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2423c036d644a9b9133288fc1b641d8; no voters:
I20260928 16:38:52.470852 18005 leader_election.cc:288] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:52.470970 18010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:52.471212 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:52.471254 18010 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [term 1 LEADER]: Becoming Leader. State: Replica: e2423c036d644a9b9133288fc1b641d8, State: Running, Role: LEADER
I20260928 16:38:52.471534 18010 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2423c036d644a9b9133288fc1b641d8" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38955 } }
I20260928 16:38:52.472162 18005 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:52.472940 18011 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2423c036d644a9b9133288fc1b641d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2423c036d644a9b9133288fc1b641d8" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38955 } } }
I20260928 16:38:52.473063 18011 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:52.473284 18011 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2423c036d644a9b9133288fc1b641d8. Latest consensus state: current_term: 1 leader_uuid: "e2423c036d644a9b9133288fc1b641d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2423c036d644a9b9133288fc1b641d8" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 38955 } } }
I20260928 16:38:52.473381 18011 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8 [sys.catalog]: This master's current role is: LEADER
W20260928 16:38:52.477155 18031 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:38:52.477272 18031 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:38:52.477393 18023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:52.478312 18023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:52.482712 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:52.473679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46095 (local address 127.27.122.254:38955)
0928 16:38:52.473804 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:52.473810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:52.474184 (+ 374us) server_negotiation.cc:407] Connection header received
0928 16:38:52.474354 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:52.474358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:52.474401 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:52.474495 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:52.475298 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:52.476058 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:52.477260 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:52.477592 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:52.477713 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:52.479340 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:52.479345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:52.479348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:52.479375 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:52.481093 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:52.481516 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:52.481521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:52.481523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:52.481611 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:52.481887 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:52.481892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:52.481894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:52.482078 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:52.482214 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:52.482447 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 16:38:52.482569 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:38:52.484866 18023 catalog_manager.cc:1347] Generated new cluster ID: 9d32c0d0735d4fb1a6b194637babfe5a
I20260928 16:38:52.484951 18023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:52.492928 18023 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:52.494064 18023 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:52.503342 18023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2423c036d644a9b9133288fc1b641d8: Generated new TSK 0
I20260928 16:38:52.504071 18023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:52.509102 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 17936
2026-09-28T16:38:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:52 dist-test-slave-g9sh krb5kdc[18036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:52 dist-test-slave-g9sh krb5kdc[18036](info): set up 2 sockets
Sep 28 16:38:52 dist-test-slave-g9sh krb5kdc[18036](info): commencing operation
krb5kdc: starting...
W20260928 16:38:53.551560 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:53 dist-test-slave-g9sh krb5kdc[18036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:38:53.672452 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:41911
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:45675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:41911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:53.780335 18049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:53.780575 18049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:53.780644 18049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:53.784602 18049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:53.784655 18049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:53.784674 18049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:53.784691 18049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:53.784735 18049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:53.789868 18049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:45675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:41911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:41911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:53.790833 18049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:53.791811 18049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:53.800563 18056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:53.800577 18055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:53.800690 18049 server_base.cc:1034] running on GCE node
W20260928 16:38:53.800828 18058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:53.801436 18049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:53.802482 18049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:53.803678 18049 hybrid_clock.cc:648] HybridClock initialized: now 1790613533803648 us; error 48 us; skew 500 ppm
Sep 28 16:38:53 dist-test-slave-g9sh krb5kdc[18036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613533, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:53.806382 18049 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:38:53.807240 18049 webserver.cc:460] Webserver started at http://127.27.122.254:39415/ using document root <none> and password file <none>
I20260928 16:38:53.807861 18049 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:53.807960 18049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:53.808182 18049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:53.810212 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "84ed010a64924055b24cafdd74afa099"
format_stamp: "Formatted at 2026-09-28 16:38:53 on dist-test-slave-g9sh"
server_key: "203d3aa7a7867c16450c755014ae080d"
server_key_iv: "ad36756d515e294eb258293342b9326a"
server_key_version: "encryptionkey@0"
I20260928 16:38:53.810774 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "84ed010a64924055b24cafdd74afa099"
format_stamp: "Formatted at 2026-09-28 16:38:53 on dist-test-slave-g9sh"
server_key: "203d3aa7a7867c16450c755014ae080d"
server_key_iv: "ad36756d515e294eb258293342b9326a"
server_key_version: "encryptionkey@0"
I20260928 16:38:53.814306 18049 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:38:53.816504 18065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:53.817709 18049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:53.817847 18049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "84ed010a64924055b24cafdd74afa099"
format_stamp: "Formatted at 2026-09-28 16:38:53 on dist-test-slave-g9sh"
server_key: "203d3aa7a7867c16450c755014ae080d"
server_key_iv: "ad36756d515e294eb258293342b9326a"
server_key_version: "encryptionkey@0"
I20260928 16:38:53.817960 18049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:53.843940 18049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:53.846830 18049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:53.847040 18049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:53.854147 18049 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:41911
I20260928 16:38:53.854167 18118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:41911 every 8 connection(s)
I20260928 16:38:53.855065 18049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:53.857750 18119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:53.858399 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 18049
I20260928 16:38:53.858546 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:53.858847 28139 external_mini_cluster.cc:1428] Setting key 0a17108d8dac563c6f265f7a3e842227
I20260928 16:38:53.864254 18119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099: Bootstrap starting.
I20260928 16:38:53.866636 18119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:53.867405 18119 log.cc:824] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:53.867558 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:53.860179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51603 (local address 127.27.122.254:41911)
0928 16:38:53.860600 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:53.860633 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:53.860667 (+ 34us) server_negotiation.cc:407] Connection header received
0928 16:38:53.861374 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:53.861393 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:53.861712 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:53.862035 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:53.862433 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:53.863382 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:53.864290 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:53.864801 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:53.864996 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:53.865827 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:53.865832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:53.865844 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:53.866210 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:53.866262 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:53.866284 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:53.866397 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:53.866537 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 16:38:53.866772 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":114,"threads_started":1}
I20260928 16:38:53.869547 18119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099: No bootstrap required, opened a new log
I20260928 16:38:53.870349 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:53.873410 18119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ed010a64924055b24cafdd74afa099" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41911 } }
I20260928 16:38:53.873661 18119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:53.873740 18119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ed010a64924055b24cafdd74afa099, State: Initialized, Role: FOLLOWER
I20260928 16:38:53.874140 18119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ed010a64924055b24cafdd74afa099" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41911 } }
I20260928 16:38:53.874290 18119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:53.874377 18119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:53.874497 18119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:53.875377 18119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ed010a64924055b24cafdd74afa099" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41911 } }
I20260928 16:38:53.875739 18119 leader_election.cc:302] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84ed010a64924055b24cafdd74afa099; no voters:
I20260928 16:38:53.876052 18119 leader_election.cc:288] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:53.876122 18129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:53.876296 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:53.872269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42473 (local address 127.27.122.254:41911)
0928 16:38:53.872374 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:53.872379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:53.872588 (+ 209us) server_negotiation.cc:407] Connection header received
0928 16:38:53.872706 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:53.872710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:53.872748 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:53.872837 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:53.873137 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:53.873681 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:53.874703 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:53.874963 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:53.875076 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:53.875811 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:53.875816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:53.875821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:53.875834 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:53.875863 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:53.875875 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:53.875964 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:53.876058 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 16:38:53.876104 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 16:38:53.876369 18129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [term 1 LEADER]: Becoming Leader. State: Replica: 84ed010a64924055b24cafdd74afa099, State: Running, Role: LEADER
I20260928 16:38:53.876647 18129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ed010a64924055b24cafdd74afa099" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41911 } }
I20260928 16:38:53.877382 18119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:53.877830 18130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84ed010a64924055b24cafdd74afa099. Latest consensus state: current_term: 1 leader_uuid: "84ed010a64924055b24cafdd74afa099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ed010a64924055b24cafdd74afa099" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41911 } } }
I20260928 16:38:53.877950 18130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:53.878180 18131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84ed010a64924055b24cafdd74afa099" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ed010a64924055b24cafdd74afa099" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 41911 } } }
I20260928 16:38:53.878324 18131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099 [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:53.878364 18133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:53.881251 18133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:53.881614 18146 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 16:38:53.886721 18133 catalog_manager.cc:1347] Generated new cluster ID: efc41aa179a84eff80ef0d41d3c11412
I20260928 16:38:53.886782 18133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:53.906167 18133 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:53.907331 18133 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:53.914407 18133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84ed010a64924055b24cafdd74afa099: Generated new TSK 0
I20260928 16:38:53.915149 18133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:53.920295 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 18049
2026-09-28T16:38:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:53 dist-test-slave-g9sh krb5kdc[18150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:53 dist-test-slave-g9sh krb5kdc[18150](info): set up 2 sockets
Sep 28 16:38:53 dist-test-slave-g9sh krb5kdc[18150](info): commencing operation
krb5kdc: starting...
W20260928 16:38:54.963994 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:55 dist-test-slave-g9sh krb5kdc[18150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:38:55.084483 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:35407
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:43181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:35407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:55.197032 18163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:55.197348 18163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:55.197443 18163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:55.201800 18163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:55.201881 18163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:55.201917 18163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:55.201957 18163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:55.201993 18163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:55.207288 18163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:43181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:35407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:35407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:55.208380 18163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:55.209496 18163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:38:55.218036 18163 server_base.cc:1034] running on GCE node
W20260928 16:38:55.217979 18169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:55.217998 18168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:55.217998 18171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:55.218837 18163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:55.219981 18163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:55.221141 18163 hybrid_clock.cc:648] HybridClock initialized: now 1790613535221128 us; error 25 us; skew 500 ppm
Sep 28 16:38:55 dist-test-slave-g9sh krb5kdc[18150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613535, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:55.223922 18163 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:38:55.224797 18163 webserver.cc:460] Webserver started at http://127.27.122.254:40205/ using document root <none> and password file <none>
I20260928 16:38:55.225384 18163 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:55.225447 18163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:55.225670 18163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:55.227684 18163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "e3807d8246cf4f95bcffbb3a5efda93a"
format_stamp: "Formatted at 2026-09-28 16:38:55 on dist-test-slave-g9sh"
server_key: "3d5b45f02fa289bb3949c2bfaa93f62f"
server_key_iv: "ce4fd1a179a1249205216c42fa9fa8ea"
server_key_version: "encryptionkey@0"
I20260928 16:38:55.228250 18163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "e3807d8246cf4f95bcffbb3a5efda93a"
format_stamp: "Formatted at 2026-09-28 16:38:55 on dist-test-slave-g9sh"
server_key: "3d5b45f02fa289bb3949c2bfaa93f62f"
server_key_iv: "ce4fd1a179a1249205216c42fa9fa8ea"
server_key_version: "encryptionkey@0"
I20260928 16:38:55.231945 18163 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:38:55.234395 18178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:55.235620 18163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:38:55.235749 18163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "e3807d8246cf4f95bcffbb3a5efda93a"
format_stamp: "Formatted at 2026-09-28 16:38:55 on dist-test-slave-g9sh"
server_key: "3d5b45f02fa289bb3949c2bfaa93f62f"
server_key_iv: "ce4fd1a179a1249205216c42fa9fa8ea"
server_key_version: "encryptionkey@0"
I20260928 16:38:55.235854 18163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:55.240345 18163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:55.243225 18163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:55.243415 18163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:55.250283 18163 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:35407
I20260928 16:38:55.250370 18231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:35407 every 8 connection(s)
I20260928 16:38:55.251199 18163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:55.253919 18232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:55.259121 18232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a: Bootstrap starting.
I20260928 16:38:55.260054 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 18163
I20260928 16:38:55.260224 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:55.260517 28139 external_mini_cluster.cc:1428] Setting key 17716fda0588a3911363e89580b9dc05
I20260928 16:38:55.262082 18232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:55.262954 18232 log.cc:824] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:55.264969 18232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a: No bootstrap required, opened a new log
I20260928 16:38:55.265673 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:55.261637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44487 (local address 127.27.122.254:35407)
0928 16:38:55.262063 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:55.262082 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:55.262126 (+ 44us) server_negotiation.cc:407] Connection header received
0928 16:38:55.262977 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:55.263001 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:55.263365 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:55.263702 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:55.264450 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:55.264454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:55.264465 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:55.264476 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:55.264523 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:55.264540 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:55.264659 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:55.264767 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 16:38:55.264940 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":107,"threads_started":1}
I20260928 16:38:55.268276 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:55.268693 18232 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3807d8246cf4f95bcffbb3a5efda93a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35407 } }
I20260928 16:38:55.269001 18232 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 16:38:55.269060 18232 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3807d8246cf4f95bcffbb3a5efda93a, State: Initialized, Role: FOLLOWER
I20260928 16:38:55.269533 18232 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3807d8246cf4f95bcffbb3a5efda93a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35407 } }
I20260928 16:38:55.269716 18232 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 16:38:55.269799 18232 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 16:38:55.269913 18232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:55.271050 18232 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3807d8246cf4f95bcffbb3a5efda93a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35407 } }
I20260928 16:38:55.271411 18232 leader_election.cc:302] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3807d8246cf4f95bcffbb3a5efda93a; no voters:
I20260928 16:38:55.271782 18232 leader_election.cc:288] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 16:38:55.271867 18242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:55.272101 18242 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [term 1 LEADER]: Becoming Leader. State: Replica: e3807d8246cf4f95bcffbb3a5efda93a, State: Running, Role: LEADER
I20260928 16:38:55.272375 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:55.270597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49753 (local address 127.27.122.254:35407)
0928 16:38:55.270699 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:55.270703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:55.270930 (+ 227us) server_negotiation.cc:407] Connection header received
0928 16:38:55.271025 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:55.271031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:55.271068 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:55.271154 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:55.271886 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:55.271891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:55.271893 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 16:38:55.271906 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:55.271933 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 16:38:55.271944 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:55.272080 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:55.272234 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 16:38:55.272271 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 16:38:55.272444 18242 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3807d8246cf4f95bcffbb3a5efda93a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35407 } }
I20260928 16:38:55.273180 18232 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:55.273931 18244 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3807d8246cf4f95bcffbb3a5efda93a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3807d8246cf4f95bcffbb3a5efda93a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35407 } } }
I20260928 16:38:55.274057 18244 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:55.274397 18243 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3807d8246cf4f95bcffbb3a5efda93a. Latest consensus state: current_term: 1 leader_uuid: "e3807d8246cf4f95bcffbb3a5efda93a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3807d8246cf4f95bcffbb3a5efda93a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 35407 } } }
I20260928 16:38:55.274470 18251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:55.274493 18243 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:55.277804 18251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:55.287535 18251 catalog_manager.cc:1347] Generated new cluster ID: ee58d022bff640559eaebe7a1e704b47
I20260928 16:38:55.287617 18251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:55.299875 18251 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:55.300213 18251 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:55.307185 18251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3807d8246cf4f95bcffbb3a5efda93a: Generated new TSK 0
I20260928 16:38:55.307926 18251 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 16:38:55.322700 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 18163
2026-09-28T16:38:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1404 ms)
[----------] 18 tests from AuthTokenIssuingTest (25158 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:55 dist-test-slave-g9sh krb5kdc[18263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:55 dist-test-slave-g9sh krb5kdc[18263](info): set up 2 sockets
Sep 28 16:38:55 dist-test-slave-g9sh krb5kdc[18263](info): commencing operation
krb5kdc: starting...
W20260928 16:38:56.364557 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:56 dist-test-slave-g9sh krb5kdc[18263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:38:56.488437 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:37827
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:37827,127.27.122.253:34573,127.27.122.252:40317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:56.596534 18276 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:38:56.596959 18276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:56.597033 18276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:56.597121 18276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:56.602312 18276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:56.602396 18276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:56.602434 18276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:56.602470 18276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:56.602514 18276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:56.608577 18276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:37827,127.27.122.253:34573,127.27.122.252:40317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:37827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:56.609694 18276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:56.610725 18276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:56.619612 18282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:56.619640 18284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:56.619590 18281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:56.619877 18276 server_base.cc:1034] running on GCE node
I20260928 16:38:56.620463 18276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:56.621601 18276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:56.622792 18276 hybrid_clock.cc:648] HybridClock initialized: now 1790613536622763 us; error 42 us; skew 500 ppm
Sep 28 16:38:56 dist-test-slave-g9sh krb5kdc[18263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613536, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:56.631083 18276 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:38:56.632042 18276 webserver.cc:460] Webserver started at http://127.27.122.254:39717/ using document root <none> and password file <none>
I20260928 16:38:56.632587 18276 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:56.632647 18276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:56.632814 18276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:56.634946 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b"
format_stamp: "Formatted at 2026-09-28 16:38:56 on dist-test-slave-g9sh"
server_key: "5addab7cbe69482ee0055f430c88cc9c"
server_key_iv: "48a18fca92e416dffdd112d3c6fade87"
server_key_version: "encryptionkey@0"
I20260928 16:38:56.635466 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b"
format_stamp: "Formatted at 2026-09-28 16:38:56 on dist-test-slave-g9sh"
server_key: "5addab7cbe69482ee0055f430c88cc9c"
server_key_iv: "48a18fca92e416dffdd112d3c6fade87"
server_key_version: "encryptionkey@0"
I20260928 16:38:56.639125 18276 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:38:56.641464 18291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:56.642711 18276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:56.642812 18276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b"
format_stamp: "Formatted at 2026-09-28 16:38:56 on dist-test-slave-g9sh"
server_key: "5addab7cbe69482ee0055f430c88cc9c"
server_key_iv: "48a18fca92e416dffdd112d3c6fade87"
server_key_version: "encryptionkey@0"
I20260928 16:38:56.642894 18276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:56.667587 18276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:56.670727 18276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:56.670884 18276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:56.678409 18276 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:37827
I20260928 16:38:56.678722 18344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:37827 every 8 connection(s)
I20260928 16:38:56.679420 18276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:56.682372 18345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:56.684139 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 18276
I20260928 16:38:56.684281 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:56.684584 28139 external_mini_cluster.cc:1428] Setting key 70f7815694436204ca2f756926a2e6b6
I20260928 16:38:56.687330 18345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.122.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.253@KRBTEST.COM" created.
Sep 28 16:38:56 dist-test-slave-g9sh krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613536, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:56.703001 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.689916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.27.122.254:37827)
0928 16:38:56.690342 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:56.690362 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:56.690680 (+ 318us) server_negotiation.cc:407] Connection header received
0928 16:38:56.691720 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:56.691724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:56.691900 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:56.692092 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:56.692974 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.693972 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.695089 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.695535 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.695729 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:56.698281 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:56.698286 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:56.698296 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:56.698317 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:56.699922 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:56.700394 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:56.700406 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:56.700408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:56.700503 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:56.700812 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:56.700816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:56.700817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:56.701245 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:56.701386 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:56.701748 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 16:38:56.701983 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":278,"thread_start_us":150,"threads_started":1}
I20260928 16:38:56.703001 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.689917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37827 (local address 127.0.0.1:54652)
0928 16:38:56.690513 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.690550 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:56.691401 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:56.692081 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:56.692086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:56.692508 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:56.692809 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:56.692823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.694095 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.694100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:56.694962 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:56.694972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.695691 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.695695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:56.695797 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:56.696452 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:56.696468 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:56.698120 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:56.700032 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:56.700043 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:56.700052 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:56.700276 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:56.700600 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:56.700603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:56.700605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:56.700714 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:56.701358 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:56.701364 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:56.701504 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 16:38:56.701676 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 16:38:56.702036 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":131,"thread_start_us":144,"threads_started":1}
I20260928 16:38:56.705494 18345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:38:56.706266 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.705984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34573 (local address 127.0.0.1:55864)
0928 16:38:56.706116 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.706204 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 16:38:56.706490 18294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34573: connect: Connection refused (error 111)
I20260928 16:38:56.707327 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.707136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34573 (local address 127.0.0.1:55880)
0928 16:38:56.707236 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.707278 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 16:38:56.707463 18345 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.253:34573: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34573: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.253.keytab.
Entry for principal kudu/127.27.122.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.253.keytab.
I20260928 16:38:56.711586 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.253:34573
--webserver_interface=127.27.122.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.253.keytab
--principal=kudu/127.27.122.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:37827,127.27.122.253:34573,127.27.122.252:40317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:38:56.756695 18345 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } attempt: 1
I20260928 16:38:56.757587 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.757294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34573 (local address 127.0.0.1:55892)
0928 16:38:56.757449 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.757511 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 16:38:56.758318 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.758097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34573 (local address 127.0.0.1:55908)
0928 16:38:56.758222 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.758269 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 16:38:56.758447 18345 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.253:34573: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34573: connect: Connection refused (error 111)
W20260928 16:38:56.821905 18352 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:38:56.822207 18352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:56.822273 18352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:56.822346 18352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:56.826311 18352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:56.826388 18352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:56.826423 18352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:56.826484 18352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:56.826519 18352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:56.826587 18345 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } attempt: 2
I20260928 16:38:56.827395 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.827081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34573 (local address 127.0.0.1:55914)
0928 16:38:56.827256 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.827323 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 16:38:56.828153 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.827917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34573 (local address 127.0.0.1:55928)
0928 16:38:56.828030 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.828084 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 16:38:56.828318 18345 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.253:34573: Network error: Client connection negotiation failed: client connection to 127.27.122.253:34573: connect: Connection refused (error 111)
I20260928 16:38:56.832020 18352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:37827,127.27.122.253:34573,127.27.122.252:40317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.253.keytab
--principal=kudu/127.27.122.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.253:34573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.122.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:56.833082 18352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:56.834054 18352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:56.842278 18358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:56.842288 18357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:56.842258 18360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:56.842660 18352 server_base.cc:1034] running on GCE node
I20260928 16:38:56.843186 18352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:56.844273 18352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:56.845460 18352 hybrid_clock.cc:648] HybridClock initialized: now 1790613536845443 us; error 29 us; skew 500 ppm
Sep 28 16:38:56 dist-test-slave-g9sh krb5kdc[18263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613536, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:56.848308 18352 init.cc:377] Logged in from keytab as kudu/127.27.122.253@KRBTEST.COM (short username kudu)
I20260928 16:38:56.849172 18352 webserver.cc:460] Webserver started at http://127.27.122.253:44601/ using document root <none> and password file <none>
I20260928 16:38:56.849670 18352 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:56.849727 18352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:56.849882 18352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:56.851765 18352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/data/instance:
uuid: "7a5f99d924fe423382ac296ee2f9b85f"
format_stamp: "Formatted at 2026-09-28 16:38:56 on dist-test-slave-g9sh"
server_key: "f3688fdd179263e0c6ec2c891d6115ef"
server_key_iv: "6dcbdb55b002f2f3b7d4a3f9dae2c8b4"
server_key_version: "encryptionkey@0"
I20260928 16:38:56.852249 18352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/wal/instance:
uuid: "7a5f99d924fe423382ac296ee2f9b85f"
format_stamp: "Formatted at 2026-09-28 16:38:56 on dist-test-slave-g9sh"
server_key: "f3688fdd179263e0c6ec2c891d6115ef"
server_key_iv: "6dcbdb55b002f2f3b7d4a3f9dae2c8b4"
server_key_version: "encryptionkey@0"
I20260928 16:38:56.855780 18352 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 16:38:56.857983 18367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:56.859103 18352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 16:38:56.859261 18352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/wal
uuid: "7a5f99d924fe423382ac296ee2f9b85f"
format_stamp: "Formatted at 2026-09-28 16:38:56 on dist-test-slave-g9sh"
server_key: "f3688fdd179263e0c6ec2c891d6115ef"
server_key_iv: "6dcbdb55b002f2f3b7d4a3f9dae2c8b4"
server_key_version: "encryptionkey@0"
I20260928 16:38:56.859361 18352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:56.880266 18352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:56.883489 18352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:56.883708 18352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:56.890537 18352 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.253:34573
I20260928 16:38:56.890858 18419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.253:34573 every 8 connection(s)
I20260928 16:38:56.891474 18352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/data/info.pb
I20260928 16:38:56.894167 18420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:56.897333 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 18352
I20260928 16:38:56.897459 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-1/wal/instance
I20260928 16:38:56.897759 28139 external_mini_cluster.cc:1428] Setting key d942a5f73db849caecc606a3374b3fc5
I20260928 16:38:56.898170 18420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.122.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.252@KRBTEST.COM" created.
Sep 28 16:38:56 dist-test-slave-g9sh krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613536, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.253@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:56.913480 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.900381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.27.122.254:37827)
0928 16:38:56.900544 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:56.900551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:56.901351 (+ 800us) server_negotiation.cc:407] Connection header received
0928 16:38:56.902347 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:56.902353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:56.902425 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:56.902539 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:56.903614 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.904496 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.905728 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.906161 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.906250 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:56.909687 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:56.909690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:56.909692 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:56.909717 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:56.911405 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:56.911892 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:56.911896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:56.911899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:56.911984 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:56.912331 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:56.912334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:56.912336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:56.912519 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:56.912631 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:56.913184 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 16:38:56.913310 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 16:38:56.914203 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.900715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37827 (local address 127.0.0.1:54656)
0928 16:38:56.901188 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.901240 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:56.902136 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:56.902682 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:56.902688 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:56.902996 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:56.903462 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:56.903475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.904634 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.904636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:56.905579 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:56.905589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.906322 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.906325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:56.906467 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:56.907026 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:56.907042 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:56.909497 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:56.911512 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:56.911517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:56.911528 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:56.911775 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:56.912080 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:56.912085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:56.912087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:56.912226 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:56.912630 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:56.912636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:56.912870 (+ 234us) client_negotiation.cc:769] Sending connection context
0928 16:38:56.913108 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 16:38:56.913421 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":145,"threads_started":1}
I20260928 16:38:56.915629 18420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:38:56.919526 18345 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } attempt: 3
Sep 28 16:38:56 dist-test-slave-g9sh krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613536, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.253@KRBTEST.COM for kudu/127.27.122.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.252.keytab.
Entry for principal kudu/127.27.122.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.252.keytab.
I20260928 16:38:56.923993 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.252:40317
--webserver_interface=127.27.122.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.252.keytab
--principal=kudu/127.27.122.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:37827,127.27.122.253:34573,127.27.122.252:40317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 16:38:56 dist-test-slave-g9sh krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613536, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for kudu/127.27.122.253@KRBTEST.COM
I20260928 16:38:56.925621 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.916135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34573 (local address 127.0.0.1:55932)
0928 16:38:56.916289 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.916303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:56.916374 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:56.916848 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:56.916850 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:56.917036 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:56.917125 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:56.917131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.918124 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.918126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:56.918877 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:56.918888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.919571 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.919575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:56.919693 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:56.920197 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:56.920235 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:56.921730 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:56.924045 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:56.924048 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:56.924049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:56.924329 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:56.924633 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:56.924635 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:56.924637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:56.924726 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:56.925224 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:56.925227 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:56.925307 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 16:38:56.925410 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 16:38:56.925516 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 16:38:56.926173 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.916250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.27.122.253:34573)
0928 16:38:56.916526 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:56.916536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:56.916553 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:38:56.916585 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:56.916589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:56.916725 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:56.916825 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:56.917228 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.918007 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.918982 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.919410 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.919512 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:56.921862 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:56.921874 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:56.921891 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:56.921915 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:56.923748 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:56.924452 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:56.924462 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:56.924464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:56.924534 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:56.924823 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:56.924826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:56.924827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:56.925109 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:56.925259 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:56.925883 (+ 624us) server_negotiation.cc:299] Negotiation successful
0928 16:38:56.926022 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":109,"threads_started":1}
I20260928 16:38:56.928208 18420 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:38:56.928983 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.928623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56444)
0928 16:38:56.928786 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.928865 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260928 16:38:56.929190 18368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
I20260928 16:38:56.929991 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.919940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34573 (local address 127.0.0.1:55946)
0928 16:38:56.920116 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.920132 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:56.920226 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:56.920556 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:56.920559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:56.920733 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:56.921364 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:56.921378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.922179 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.922202 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:56.922897 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:56.922910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.923537 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.923541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:56.923649 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:56.924057 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:56.924074 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:56.925673 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:56.928293 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:56.928299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:56.928304 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:56.928576 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:56.928884 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:56.928890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:56.928895 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:56.929028 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:56.929533 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:56.929536 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:56.929642 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 16:38:56.929759 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 16:38:56.929882 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 16:38:56.930239 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.920013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.27.122.253:34573)
0928 16:38:56.920368 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:56.920375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:56.920385 (+ 10us) server_negotiation.cc:407] Connection header received
0928 16:38:56.920410 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:56.920411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:56.920441 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:56.920542 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:56.921490 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.922054 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.923032 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:56.923398 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:56.923513 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:56.925846 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:56.925852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:56.925856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:56.925927 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:56.928175 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:56.928683 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:56.928686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:56.928688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:56.928773 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:56.929155 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:56.929159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:56.929160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:56.929411 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:56.929524 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:56.929862 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 16:38:56.929992 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"spinlock_wait_cycles":4352,"thread_start_us":118,"threads_started":1}
I20260928 16:38:56.930860 18345 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:38:56.931355 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.930783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56458)
0928 16:38:56.931242 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.931291 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376}
W20260928 16:38:56.931546 18420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:40317: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
I20260928 16:38:56.931597 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.931291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56468)
0928 16:38:56.931446 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.931512 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 16:38:56.932329 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.932005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56480)
0928 16:38:56.932166 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.932228 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260928 16:38:56.932485 18345 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:40317: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
I20260928 16:38:56.963645 18345 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } attempt: 1
I20260928 16:38:56.964422 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.964103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56484)
0928 16:38:56.964261 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.964329 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 16:38:56.965173 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.964961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56492)
0928 16:38:56.965054 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.965106 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 16:38:56.965313 18345 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:40317: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
I20260928 16:38:56.980794 18420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } attempt: 1
I20260928 16:38:56.981552 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.981304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56500)
0928 16:38:56.981433 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.981486 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 16:38:56.982256 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:56.982045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56516)
0928 16:38:56.982151 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:56.982204 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260928 16:38:56.982380 18420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:40317: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
W20260928 16:38:57.036541 18427 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:38:57.036835 18427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:57.036903 18427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:57.037024 18427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 16:38:57.040647 18345 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } attempt: 2
W20260928 16:38:57.041405 18427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:57.041464 18427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260928 16:38:57.041479 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.041141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56530)
0928 16:38:57.041304 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:57.041375 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 16:38:57.041496 18427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:57.041522 18427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:57.041546 18427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:57.042201 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.042002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56540)
0928 16:38:57.042101 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:57.042146 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 16:38:57.042369 18345 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:40317: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
I20260928 16:38:57.047073 18427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:37827,127.27.122.253:34573,127.27.122.252:40317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.252.keytab
--principal=kudu/127.27.122.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.252:40317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.122.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:57.048044 18427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:57.049101 18427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 16:38:57.050529 18420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } attempt: 2
I20260928 16:38:57.051245 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.051001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56544)
0928 16:38:57.051124 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:57.051173 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 16:38:57.051947 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.051772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56556)
0928 16:38:57.051862 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:57.051901 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260928 16:38:57.052106 18420 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:40317: Network error: Client connection negotiation failed: client connection to 127.27.122.252:40317: connect: Connection refused (error 111)
W20260928 16:38:57.057722 18433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:57.057722 18434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:57.057811 18436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:57.057933 18427 server_base.cc:1034] running on GCE node
I20260928 16:38:57.058560 18427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:57.059625 18427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:57.060817 18427 hybrid_clock.cc:648] HybridClock initialized: now 1790613537060805 us; error 25 us; skew 500 ppm
Sep 28 16:38:57 dist-test-slave-g9sh krb5kdc[18263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613537, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:57.063571 18427 init.cc:377] Logged in from keytab as kudu/127.27.122.252@KRBTEST.COM (short username kudu)
I20260928 16:38:57.064421 18427 webserver.cc:460] Webserver started at http://127.27.122.252:44537/ using document root <none> and password file <none>
I20260928 16:38:57.064962 18427 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:57.065027 18427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:57.065199 18427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:57.067055 18427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/data/instance:
uuid: "5762015b2a604dcb9af6f80a567e311d"
format_stamp: "Formatted at 2026-09-28 16:38:57 on dist-test-slave-g9sh"
server_key: "631c190eb4ba9a79d3cb4bbf7bf9b882"
server_key_iv: "57deaffd5d00e2320274ad97906d7fc4"
server_key_version: "encryptionkey@0"
I20260928 16:38:57.067538 18427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/wal/instance:
uuid: "5762015b2a604dcb9af6f80a567e311d"
format_stamp: "Formatted at 2026-09-28 16:38:57 on dist-test-slave-g9sh"
server_key: "631c190eb4ba9a79d3cb4bbf7bf9b882"
server_key_iv: "57deaffd5d00e2320274ad97906d7fc4"
server_key_version: "encryptionkey@0"
I20260928 16:38:57.070976 18427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:38:57.073319 18443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:57.074538 18427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:57.074676 18427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/wal
uuid: "5762015b2a604dcb9af6f80a567e311d"
format_stamp: "Formatted at 2026-09-28 16:38:57 on dist-test-slave-g9sh"
server_key: "631c190eb4ba9a79d3cb4bbf7bf9b882"
server_key_iv: "57deaffd5d00e2320274ad97906d7fc4"
server_key_version: "encryptionkey@0"
I20260928 16:38:57.074783 18427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:57.101698 18427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:57.104617 18427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:57.104821 18427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:57.111817 18427 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.252:40317
I20260928 16:38:57.112092 18495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.252:40317 every 8 connection(s)
I20260928 16:38:57.112847 18427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/data/info.pb
I20260928 16:38:57.115492 18496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:57.119063 18496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:38:57.121999 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 18427
I20260928 16:38:57.122145 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790613255639823-28139-0/minicluster-data/master-2/wal/instance
I20260928 16:38:57.122421 28139 external_mini_cluster.cc:1428] Setting key 493633249e90b053f9e1619551d392a8
Sep 28 16:38:57 dist-test-slave-g9sh krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613537, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.252@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
Sep 28 16:38:57 dist-test-slave-g9sh krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.252@KRBTEST.COM
I20260928 16:38:57.133131 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.120850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54664 (local address 127.27.122.254:37827)
0928 16:38:57.121063 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.121067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.121950 (+ 883us) server_negotiation.cc:407] Connection header received
0928 16:38:57.123144 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.123147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.123208 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.123291 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.124450 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.125019 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.126245 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.126476 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.126592 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.129272 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:57.129275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:57.129279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:57.129311 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:57.131008 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.131547 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.131551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.131552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.131631 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.131972 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.131976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.131978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.132181 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:57.132293 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:57.132793 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 16:38:57.132957 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260928 16:38:57.133931 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.121277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:37827 (local address 127.0.0.1:54664)
0928 16:38:57.121721 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:57.121821 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:57.122896 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:57.123523 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:57.123530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:57.123856 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:57.124300 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:57.124312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.125131 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.125136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:57.126113 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:57.126123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.126628 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.126633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:57.126821 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.127410 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:57.127434 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:57.129127 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:57.131124 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:57.131135 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:57.131145 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:57.131437 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:57.131727 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:57.131731 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:57.131733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:57.131871 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:57.132291 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:57.132298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:57.132543 (+ 245us) client_negotiation.cc:769] Sending connection context
0928 16:38:57.132719 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 16:38:57.133053 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":187,"thread_start_us":100,"threads_started":1}
I20260928 16:38:57.133939 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.123092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.27.122.252:40317)
0928 16:38:57.123692 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.123700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.123718 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:38:57.123750 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.123754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.123903 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.124071 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.124804 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.125716 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.126565 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.126939 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.127061 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.129792 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:57.129797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:57.129806 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:57.129827 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:57.131400 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.131894 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.131904 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.131905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.131979 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.132267 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.132271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.132273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.132532 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:57.132748 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:57.133012 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 16:38:57.133163 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":522,"thread_start_us":161,"threads_started":1}
I20260928 16:38:57.135166 18496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:38:57 dist-test-slave-g9sh krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613537, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.252@KRBTEST.COM for kudu/127.27.122.253@KRBTEST.COM
I20260928 16:38:57.141646 18345 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } attempt: 3
Sep 28 16:38:57 dist-test-slave-g9sh krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:57.143286 18420 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } attempt: 3
I20260928 16:38:57.145272 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.135744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55960 (local address 127.27.122.253:34573)
0928 16:38:57.135867 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.135871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.135882 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:38:57.135918 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.135920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.135958 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.136045 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.136721 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.137581 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.138614 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.138991 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.139098 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.141421 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:57.141424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:57.141426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:57.141450 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:57.143220 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.143758 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.143764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.143768 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.143869 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.144223 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.144227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.144229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.144487 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:57.144559 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:57.144975 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 16:38:57.145115 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:38:57.145691 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.135658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:34573 (local address 127.0.0.1:55960)
0928 16:38:57.135743 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:57.135759 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:57.135826 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:57.136279 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:57.136285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:57.136455 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:57.136572 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:57.136580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.137690 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.137693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:57.138499 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:57.138508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.139107 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.139109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:57.139192 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.139649 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:57.139664 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:57.141260 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:57.143360 (+ 2100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:57.143363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:57.143365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:57.143640 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:57.143993 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:57.143999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:57.144001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:57.144103 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:57.144634 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:57.144637 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:57.144773 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 16:38:57.145404 (+ 631us) client_negotiation.cc:240] Negotiation successful
0928 16:38:57.145531 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 16:38:57.146847 18496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:38:57 dist-test-slave-g9sh krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613536, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for kudu/127.27.122.252@KRBTEST.COM
I20260928 16:38:57.148624 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.136493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.27.122.254:37827)
0928 16:38:57.136617 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.136624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.136640 (+ 16us) server_negotiation.cc:407] Connection header received
0928 16:38:57.136753 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.136755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.136788 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.136843 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.137563 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.138325 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.139649 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.139984 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.140112 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.143542 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:57.143547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:57.143554 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:57.143593 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:57.145506 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.146773 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.146776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.146780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.146882 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.147210 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.147213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.147215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.147441 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:57.147662 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:57.148314 (+ 652us) server_negotiation.cc:299] Negotiation successful
0928 16:38:57.148426 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 16:38:57 dist-test-slave-g9sh krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613536, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.253@KRBTEST.COM for kudu/127.27.122.252@KRBTEST.COM
I20260928 16:38:57.152441 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.142320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56576)
0928 16:38:57.142680 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:57.142692 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:57.142761 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:57.143039 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:57.143043 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:57.143222 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:57.143332 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:57.143339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.144485 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.144489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:57.145264 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:57.145276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.146191 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.146194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:57.146277 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.146743 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:57.146757 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:57.148196 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:57.150418 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:57.150421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:57.150423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:57.150682 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:57.151000 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:57.151005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:57.151009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:57.151104 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:57.151540 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:57.151546 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:57.151646 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 16:38:57.152143 (+ 497us) client_negotiation.cc:240] Negotiation successful
0928 16:38:57.152272 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293}
I20260928 16:38:57.153071 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.142403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.27.122.252:40317)
0928 16:38:57.142510 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.142513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.142760 (+ 247us) server_negotiation.cc:407] Connection header received
0928 16:38:57.142849 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.142853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.142897 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.142983 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.143492 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.144319 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.145664 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.146069 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.146167 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.148335 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:57.148338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:57.148341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:57.148362 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:57.150305 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.150799 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.150801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.150803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.150884 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.151233 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.151236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.151237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.151428 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:57.152712 (+ 1284us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:57.152786 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 16:38:57.152932 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 28 16:38:57 dist-test-slave-g9sh krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613537, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.252@KRBTEST.COM for kudu/127.27.122.252@KRBTEST.COM
I20260928 16:38:57.155678 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.143735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56584)
0928 16:38:57.144247 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:57.144262 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:57.144346 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:57.145799 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:57.145804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:57.145975 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:57.146066 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:57.146072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.147812 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.147816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:57.148610 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:57.148623 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.149334 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.149337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:57.149422 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.149851 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:57.149864 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:57.151270 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:57.153927 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:57.153930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:57.153932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:57.154199 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:57.154524 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:57.154528 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:57.154532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:57.154625 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:57.155130 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:57.155137 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:57.155245 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 16:38:57.155343 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 16:38:57.155480 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428}
Sep 28 16:38:57 dist-test-slave-g9sh krb5kdc[18263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.253@KRBTEST.COM
I20260928 16:38:57.156399 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.144804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.27.122.252:40317)
0928 16:38:57.145536 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.145545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.145556 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:38:57.145600 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.145603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.145640 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.146197 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.146282 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.147095 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.148764 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.149201 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.149624 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.151412 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:57.151419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:57.151424 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:57.151454 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:57.153777 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.154324 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.154327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.154329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.154412 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.154747 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.154753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.154755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.154982 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:57.155954 (+ 972us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:57.156059 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 16:38:57.156207 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":172,"threads_started":1}
I20260928 16:38:57.156680 18345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b: Bootstrap starting.
I20260928 16:38:57.159053 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.147346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:40317 (local address 127.0.0.1:56590)
0928 16:38:57.147431 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:57.147445 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:57.147513 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:57.148068 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:57.148082 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:57.148239 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:57.148313 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:57.148319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.150013 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.150017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:57.150806 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:57.150818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.151478 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.151481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:57.151570 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.152196 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:57.152208 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:57.154440 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:57.157638 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:57.157641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:57.157643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:57.157869 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:57.158139 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:57.158142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:57.158144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:57.158227 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:57.158640 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:57.158643 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:57.158720 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 16:38:57.158799 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 16:38:57.158911 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 16:38:57.159492 18420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f: Bootstrap starting.
I20260928 16:38:57.159825 18345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:57.159986 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.147535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.27.122.252:40317)
0928 16:38:57.147813 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.147822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.147833 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:38:57.147863 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.147865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.147959 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.148401 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.148483 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.149895 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.150934 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.151381 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.151744 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.154601 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:57.154604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:57.154606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:57.154627 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:57.157272 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.157967 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.157969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.157971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.158039 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.158328 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.158330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.158331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.158523 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:57.159528 (+ 1005us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:57.159670 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 16:38:57.159777 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":138,"threads_started":1}
I20260928 16:38:57.159986 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.149973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.27.122.253:34573)
0928 16:38:57.150096 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.150102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.150354 (+ 252us) server_negotiation.cc:407] Connection header received
0928 16:38:57.150530 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.150533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.150578 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.150632 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.151509 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.152261 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.153357 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.153671 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.153746 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.156510 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:57.156513 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:57.156516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:57.156541 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:57.158156 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.158625 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.158628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.158630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.158708 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.158996 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.158999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.159001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.159229 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:57.159379 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:57.159740 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 16:38:57.159854 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 16:38:57.160763 18345 log.cc:824] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:57.162020 18420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:57.162880 18420 log.cc:824] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:57.163308 18496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d: Bootstrap starting.
I20260928 16:38:57.163820 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:38:57.165025 18420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f: No bootstrap required, opened a new log
I20260928 16:38:57.165679 18496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d: Neither blocks nor log segments found. Creating new log.
I20260928 16:38:57.166287 18496 log.cc:824] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d: Log is configured to *not* fsync() on all Append() calls
I20260928 16:38:57.165246 18318 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54674
I20260928 16:38:57.167903 18496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d: No bootstrap required, opened a new log
I20260928 16:38:57.169090 18420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } }
I20260928 16:38:57.169325 18420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5f99d924fe423382ac296ee2f9b85f, State: Initialized, Role: FOLLOWER
I20260928 16:38:57.169669 18420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } }
I20260928 16:38:57.170967 18420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:57.171080 18345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b: No bootstrap required, opened a new log
I20260928 16:38:57.172065 18496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } }
I20260928 16:38:57.172286 18496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5762015b2a604dcb9af6f80a567e311d, State: Initialized, Role: FOLLOWER
I20260928 16:38:57.172585 18496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } }
I20260928 16:38:57.173066 18506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } } }
I20260928 16:38:57.173205 18506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:38:57.173806 18496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:38:57.174211 18345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } }
I20260928 16:38:57.174425 18345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e8ac6a3fade4fe486b6f98ea6941d9b, State: Initialized, Role: FOLLOWER
I20260928 16:38:57.174787 18345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } }
I20260928 16:38:57.176115 18345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [sys.catalog]: configured and running, proceeding with master startup.
W20260928 16:38:57.178059 18518 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:38:57.178294 18518 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:38:57.175932 18352 master_runner.cc:186] Error getting master registration for 127.27.122.254:37827: OK, instance_id { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" instance_seqno: 1790613536676381 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:38:57.179200 18352 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:38:57.179489 18318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 16:38:57.179610 18318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:57.180869 18318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } }
I20260928 16:38:57.181882 18318 leader_election.cc:288] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [CANDIDATE]: Term 1 election: Requested vote from peers 7a5f99d924fe423382ac296ee2f9b85f (127.27.122.253:34573), 5762015b2a604dcb9af6f80a567e311d (127.27.122.252:40317)
I20260928 16:38:57.182583 18470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5762015b2a604dcb9af6f80a567e311d"
I20260928 16:38:57.182804 18470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:57.182801 18393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7a5f99d924fe423382ac296ee2f9b85f"
I20260928 16:38:57.182994 18393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:38:57.183864 18470 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e8ac6a3fade4fe486b6f98ea6941d9b in term 1.
I20260928 16:38:57.184243 18393 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e8ac6a3fade4fe486b6f98ea6941d9b in term 1.
I20260928 16:38:57.184290 18292 leader_election.cc:302] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e8ac6a3fade4fe486b6f98ea6941d9b, 5762015b2a604dcb9af6f80a567e311d; no voters:
I20260928 16:38:57.178476 18512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } } }
I20260928 16:38:57.186012 18512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d [sys.catalog]: This master's current role is: FOLLOWER
W20260928 16:38:57.179615 18530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:38:57.186112 18530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:38:57.187669 18276 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260928 16:38:57.187783 18276 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:38:57.187844 18523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } } }
I20260928 16:38:57.187979 18523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:38:57.188050 18523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:38:57.188108 18523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [term 1 LEADER]: Becoming Leader. State: Replica: 0e8ac6a3fade4fe486b6f98ea6941d9b, State: Running, Role: LEADER
I20260928 16:38:57.188264 18427 master_runner.cc:263] No leader master found from master 5762015b2a604dcb9af6f80a567e311d
I20260928 16:38:57.188334 18427 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:38:57.188580 18523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } }
W20260928 16:38:57.190243 18543 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:38:57.190315 18543 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:38:57.190373 18523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e8ac6a3fade4fe486b6f98ea6941d9b. Latest consensus state: current_term: 1 leader_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } } }
I20260928 16:38:57.190465 18523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:57.201371 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.189887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54678 (local address 127.27.122.254:37827)
0928 16:38:57.189995 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.189999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.191008 (+ 1009us) server_negotiation.cc:407] Connection header received
0928 16:38:57.191195 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.191198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.191241 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.191294 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.192174 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.192978 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.194425 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.194769 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.194844 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.197250 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:57.197253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:57.197256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:57.197282 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:57.199198 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.199820 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.199823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.199825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.200032 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.200362 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.200365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.200367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.200587 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:57.200697 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:57.201112 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 16:38:57.201225 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 16:38:57.202266 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.189369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.27.122.253:34573)
0928 16:38:57.189482 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.189486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.190750 (+ 1264us) server_negotiation.cc:407] Connection header received
0928 16:38:57.190903 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.190907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.190960 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.191019 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.192139 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.192971 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.194425 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.194769 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.194844 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.197254 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:57.197257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:57.197259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:57.197282 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:57.199313 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.199820 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.199823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.199825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.199920 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.200373 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.200377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.200378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.200587 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:57.200689 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:57.202000 (+ 1311us) server_negotiation.cc:299] Negotiation successful
0928 16:38:57.202131 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 16:38:57.203249 18553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:38:57.204155 18553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:38:57.209712 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:57.190509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.27.122.252:40317)
0928 16:38:57.190626 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.190631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.201155 (+ 10524us) server_negotiation.cc:407] Connection header received
0928 16:38:57.201334 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.201337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.201382 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.201435 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.202363 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.203179 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.204339 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:57.204689 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:57.204795 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:57.206240 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:57.206243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:57.206246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:57.206271 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:57.207740 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.208158 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.208161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.208163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.208285 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:57.208551 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:57.208554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:57.208556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:57.208727 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:57.208879 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:57.209437 (+ 558us) server_negotiation.cc:299] Negotiation successful
0928 16:38:57.209544 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 16:38:57.219267 18470 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d [term 1 FOLLOWER]: Refusing update from remote peer 0e8ac6a3fade4fe486b6f98ea6941d9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:57.220006 18544 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:38:57.223680 18512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e8ac6a3fade4fe486b6f98ea6941d9b. Latest consensus state: current_term: 1 leader_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } } }
I20260928 16:38:57.223784 18512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:38:57.224761 18393 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f [term 1 FOLLOWER]: Refusing update from remote peer 0e8ac6a3fade4fe486b6f98ea6941d9b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:38:57.225386 18544 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:38:57.229111 18506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e8ac6a3fade4fe486b6f98ea6941d9b. Latest consensus state: current_term: 1 leader_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } } }
I20260928 16:38:57.229216 18506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:38:57.232677 18544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } } }
I20260928 16:38:57.232803 18544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:57.234279 18512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } } }
I20260928 16:38:57.234385 18512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:38:57.234990 18555 mvcc.cc:205] Tried to move back new op lower bound from 7334353048443715584 to 7334353048327958528. Current Snapshot: MvccSnapshot[applied={T|T < 7334353048443715584}]
I20260928 16:38:57.235348 18556 mvcc.cc:205] Tried to move back new op lower bound from 7334353048443715584 to 7334353048327958528. Current Snapshot: MvccSnapshot[applied={T|T < 7334353048443715584}]
I20260928 16:38:57.241245 18523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } } }
I20260928 16:38:57.241357 18523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b [sys.catalog]: This master's current role is: LEADER
I20260928 16:38:57.242427 18506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e8ac6a3fade4fe486b6f98ea6941d9b" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 37827 } } peers { permanent_uuid: "7a5f99d924fe423382ac296ee2f9b85f" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 34573 } } peers { permanent_uuid: "5762015b2a604dcb9af6f80a567e311d" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 40317 } } }
I20260928 16:38:57.242537 18506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:38:57.243338 18558 mvcc.cc:205] Tried to move back new op lower bound from 7334353048443715584 to 7334353048327958528. Current Snapshot: MvccSnapshot[applied={T|T < 7334353048443715584}]
I20260928 16:38:57.243862 18553 catalog_manager.cc:1347] Generated new cluster ID: a9981467ea6f4457b4b502ddf98c9911
I20260928 16:38:57.243923 18553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:38:57.265234 18553 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:38:57.268344 18553 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:38:57.283280 18553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e8ac6a3fade4fe486b6f98ea6941d9b: Generated new TSK 0
I20260928 16:38:57.284309 18553 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:38:57.315007 18426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:57.312066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55984 (local address 127.27.122.253:34573)
0928 16:38:57.312193 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.312197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.313970 (+ 1773us) server_negotiation.cc:407] Connection header received
0928 16:38:57.314168 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.314171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.314222 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.314273 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.314854 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55984: BlockingRecv error: recv got EOF from 127.0.0.1:55984 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 16:38:57.316349 18348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:57.312791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.27.122.254:37827)
0928 16:38:57.312918 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.312922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.315504 (+ 2582us) server_negotiation.cc:407] Connection header received
0928 16:38:57.315667 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.315670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.315717 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.315765 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.316240 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54688: BlockingRecv error: recv got EOF from 127.0.0.1:54688 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 16:38:57.317754 18500 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:57.313453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.27.122.252:40317)
0928 16:38:57.313560 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.313565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.316800 (+ 3235us) server_negotiation.cc:407] Connection header received
0928 16:38:57.317075 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.317078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.317125 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.317172 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.317642 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56606: BlockingRecv error: recv got EOF from 127.0.0.1:56606 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260928 16:38:57.321962 18426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:57.320473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56000 (local address 127.27.122.253:34573)
0928 16:38:57.320570 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.320574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.321088 (+ 514us) server_negotiation.cc:407] Connection header received
0928 16:38:57.321285 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.321288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.321356 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.321405 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.321860 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56000: BlockingRecv error: recv got EOF from 127.0.0.1:56000 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 16:38:57.326463 18426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:57.325117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56014 (local address 127.27.122.253:34573)
0928 16:38:57.325207 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.325211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.325687 (+ 476us) server_negotiation.cc:407] Connection header received
0928 16:38:57.325845 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.325848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.325890 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.325935 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.326375 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56014: BlockingRecv error: recv got EOF from 127.0.0.1:56014 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260928 16:38:57.334296 18426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:57.332652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56026 (local address 127.27.122.253:34573)
0928 16:38:57.332753 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.332757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.333286 (+ 529us) server_negotiation.cc:407] Connection header received
0928 16:38:57.333477 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.333480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.333525 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.333641 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.334200 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56026: BlockingRecv error: recv got EOF from 127.0.0.1:56026 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 16:38:57.342018 18426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:57.340484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.27.122.253:34573)
0928 16:38:57.340610 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.340614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.341178 (+ 564us) server_negotiation.cc:407] Connection header received
0928 16:38:57.341351 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.341353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.341396 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.341443 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.341915 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56030: BlockingRecv error: recv got EOF from 127.0.0.1:56030 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 16:38:57.353725 18426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:57.352138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.27.122.253:34573)
0928 16:38:57.352237 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.352241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.352769 (+ 528us) server_negotiation.cc:407] Connection header received
0928 16:38:57.353032 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.353035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.353085 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.353138 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.353633 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56038: BlockingRecv error: recv got EOF from 127.0.0.1:56038 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 16:38:57.373662 18426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:57.371915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.27.122.253:34573)
0928 16:38:57.372058 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.372063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.372630 (+ 567us) server_negotiation.cc:407] Connection header received
0928 16:38:57.372839 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.372843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.372890 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.372982 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.373554 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56044: BlockingRecv error: recv got EOF from 127.0.0.1:56044 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 16:38:57.409693 18426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:57.408291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.27.122.253:34573)
0928 16:38:57.408432 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.408437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.408710 (+ 273us) server_negotiation.cc:407] Connection header received
0928 16:38:57.408884 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.408888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.408996 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.409091 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.409572 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56056: BlockingRecv error: recv got EOF from 127.0.0.1:56056 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 16:38:57.477347 18426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:57.476122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.27.122.253:34573)
0928 16:38:57.476268 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.476273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.476396 (+ 123us) server_negotiation.cc:407] Connection header received
0928 16:38:57.476558 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.476561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.476609 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.476691 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.477199 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56058: BlockingRecv error: recv got EOF from 127.0.0.1:56058 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 16:38:57.608633 18426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:57.607338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.27.122.253:34573)
0928 16:38:57.607489 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.607493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.607650 (+ 157us) server_negotiation.cc:407] Connection header received
0928 16:38:57.607779 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.607781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.607829 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.607911 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.608460 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56060: BlockingRecv error: recv got EOF from 127.0.0.1:56060 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 16:38:57.867782 18426 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:38:57.866615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.27.122.253:34573)
0928 16:38:57.866773 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:57.866786 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:57.866858 (+ 72us) server_negotiation.cc:407] Connection header received
0928 16:38:57.866981 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:57.866985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:57.867046 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:57.867135 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:57.867641 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56064: BlockingRecv error: recv got EOF from 127.0.0.1:56064 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 16:38:58.179733 18518 catalog_manager.cc:1259] Loaded cluster ID: a9981467ea6f4457b4b502ddf98c9911
I20260928 16:38:58.179811 18518 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f: loading cluster ID for follower catalog manager: success
I20260928 16:38:58.181531 18518 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f: acquiring CA information for follower catalog manager: success
I20260928 16:38:58.182636 18518 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a5f99d924fe423382ac296ee2f9b85f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:38:58.187619 18530 catalog_manager.cc:1259] Loaded cluster ID: a9981467ea6f4457b4b502ddf98c9911
I20260928 16:38:58.187673 18530 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d: loading cluster ID for follower catalog manager: success
I20260928 16:38:58.189234 18530 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d: acquiring CA information for follower catalog manager: success
I20260928 16:38:58.190306 18530 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5762015b2a604dcb9af6f80a567e311d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:38:58.385990 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:58.381960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.27.122.253:34573)
0928 16:38:58.382229 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:58.382233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:58.382296 (+ 63us) server_negotiation.cc:407] Connection header received
0928 16:38:58.382449 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:58.382452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:58.382604 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:58.382706 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:38:58.382994 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:58.383530 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:58.384395 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:58.384645 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:58.384729 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:58.385082 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:38:58.385533 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:38:58.385653 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:58.385725 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 16:38:58.385776 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
I20260928 16:38:58.392484 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:58.389036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.27.122.252:40317)
0928 16:38:58.389308 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:58.389311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:58.389324 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:38:58.389419 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:58.389421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:58.389564 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:58.389660 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:38:58.389898 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:58.390399 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:58.391219 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:58.391460 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:58.391558 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:58.391792 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:38:58.392179 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:38:58.392271 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:58.392324 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 16:38:58.392372 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260928 16:38:58.399467 18633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:58.395654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.27.122.252:40317)
0928 16:38:58.395754 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:58.395759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:58.395806 (+ 47us) server_negotiation.cc:407] Connection header received
0928 16:38:58.395940 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:58.395943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:58.396000 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:58.396069 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:38:58.396331 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:58.397141 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:58.398173 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:58.398405 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:58.398522 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:58.398822 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:38:58.398891 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:38:58.398959 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:58.399270 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 16:38:58.399339 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 16:38:58.400069 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:58.395802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.27.122.253:34573)
0928 16:38:58.395929 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:58.395933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:58.396178 (+ 245us) server_negotiation.cc:407] Connection header received
0928 16:38:58.396272 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:58.396275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:58.396320 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:58.396411 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:38:58.396699 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:58.397447 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:58.398762 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:58.399089 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:58.399204 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:58.399542 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:38:58.399617 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:38:58.399686 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:58.399836 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 16:38:58.399898 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:38:58.400491 18639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:58.395409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.27.122.254:37827)
0928 16:38:58.396086 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:58.396090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:58.396105 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:38:58.396201 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:58.396205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:58.396355 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:58.396590 (+ 235us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:38:58.396831 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:58.397591 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:58.398803 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:58.399041 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:58.399150 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:58.399526 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:38:58.399851 (+ 325us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:38:58.399937 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:58.400310 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 16:38:58.400376 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":74,"threads_started":1}
I20260928 16:38:58.403607 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 18276
I20260928 16:38:58.410466 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 18352
I20260928 16:38:58.416226 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 18427
2026-09-28T16:38:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3092 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 16:38:58 dist-test-slave-g9sh krb5kdc[18644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 16:38:58 dist-test-slave-g9sh krb5kdc[18644](info): set up 2 sockets
Sep 28 16:38:58 dist-test-slave-g9sh krb5kdc[18644](info): commencing operation
krb5kdc: starting...
W20260928 16:38:59.459537 28139 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/test-user.keytab.
Sep 28 16:38:59 dist-test-slave-g9sh krb5kdc[18644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T16:38:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T16:38:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.122.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
Entry for principal kudu/127.27.122.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab.
I20260928 16:38:59.580037 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:46159
--webserver_interface=127.27.122.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:46159,127.27.122.253:36151,127.27.122.252:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 16:38:59.699533 18657 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:38:59.699790 18657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:59.699813 18657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:59.699915 18657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:59.703848 18657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:59.703897 18657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:59.703917 18657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:59.703933 18657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:59.703946 18657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:59.709100 18657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:46159,127.27.122.253:36151,127.27.122.252:44325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.254.keytab
--principal=kudu/127.27.122.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.254:46159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.122.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:59.710117 18657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:59.711158 18657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:59.719900 18663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:59.719887 18662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:59.719904 18665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:59.720023 18657 server_base.cc:1034] running on GCE node
I20260928 16:38:59.721606 18657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:59.722589 18657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:59.723747 18657 hybrid_clock.cc:648] HybridClock initialized: now 1790613539723735 us; error 25 us; skew 500 ppm
Sep 28 16:38:59 dist-test-slave-g9sh krb5kdc[18644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:59.726604 18657 init.cc:377] Logged in from keytab as kudu/127.27.122.254@KRBTEST.COM (short username kudu)
I20260928 16:38:59.727531 18657 webserver.cc:460] Webserver started at http://127.27.122.254:34589/ using document root <none> and password file <none>
I20260928 16:38:59.728089 18657 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:59.728149 18657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:59.728394 18657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:59.730482 18657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/data/instance:
uuid: "cbc53ea61be4449487012389f0e7359a"
format_stamp: "Formatted at 2026-09-28 16:38:59 on dist-test-slave-g9sh"
server_key: "7ccc9fe6e6d58ae3714b83c8a470d345"
server_key_iv: "c1eee194dd6ec851dac98c7045fbbcfc"
server_key_version: "encryptionkey@0"
I20260928 16:38:59.731068 18657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/wal/instance:
uuid: "cbc53ea61be4449487012389f0e7359a"
format_stamp: "Formatted at 2026-09-28 16:38:59 on dist-test-slave-g9sh"
server_key: "7ccc9fe6e6d58ae3714b83c8a470d345"
server_key_iv: "c1eee194dd6ec851dac98c7045fbbcfc"
server_key_version: "encryptionkey@0"
I20260928 16:38:59.734735 18657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 16:38:59.737097 18672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:59.738313 18657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:38:59.738467 18657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/wal
uuid: "cbc53ea61be4449487012389f0e7359a"
format_stamp: "Formatted at 2026-09-28 16:38:59 on dist-test-slave-g9sh"
server_key: "7ccc9fe6e6d58ae3714b83c8a470d345"
server_key_iv: "c1eee194dd6ec851dac98c7045fbbcfc"
server_key_version: "encryptionkey@0"
I20260928 16:38:59.738605 18657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:59.762377 18657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:59.765046 18657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:59.765249 18657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:59.772039 18657 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.254:46159
I20260928 16:38:59.772311 18724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.254:46159 every 8 connection(s)
I20260928 16:38:59.772989 18657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/data/info.pb
I20260928 16:38:59.775624 18725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:59.775827 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 18657
I20260928 16:38:59.775930 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-0/wal/instance
I20260928 16:38:59.776237 28139 external_mini_cluster.cc:1428] Setting key 56e6b5ccccffa0c95b61a9e28e5af96f
I20260928 16:38:59.780304 18725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.122.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.253@KRBTEST.COM" created.
Sep 28 16:38:59 dist-test-slave-g9sh krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:59.796866 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:59.783030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:46159 (local address 127.0.0.1:57954)
0928 16:38:59.783631 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:59.783680 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:59.784486 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:59.785369 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:59.785374 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:59.785684 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:59.786068 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:59.786084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:59.787200 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:59.787203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:59.788397 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:59.788412 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:59.789111 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:59.789115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:59.789313 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:59.789908 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:59.789925 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:59.791724 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:59.793828 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:59.793839 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:59.793851 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:59.794192 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:59.794538 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:59.794544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:59.794547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:59.794700 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:59.795310 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:59.795320 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:59.795503 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 16:38:59.795695 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 16:38:59.796150 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":113,"spinlock_wait_cycles":52480,"thread_start_us":140,"threads_started":1}
I20260928 16:38:59.796886 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:59.783032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.27.122.254:46159)
0928 16:38:59.783523 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:59.783542 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:59.783805 (+ 263us) server_negotiation.cc:407] Connection header received
0928 16:38:59.784813 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:59.784823 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:59.785149 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:59.785307 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:59.786222 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:59.787075 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:59.788533 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:59.788952 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:59.789139 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:59.791914 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:59.791923 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:59.791941 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:59.791965 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:59.793711 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:59.794311 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:59.794316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:59.794318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:59.794422 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:59.794801 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:59.794803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:59.794804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:59.795161 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:59.795255 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:59.795808 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 16:38:59.796035 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":327,"thread_start_us":140,"threads_started":1}
I20260928 16:38:59.799543 18725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:38:59.800241 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:59.799999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:36151 (local address 127.0.0.1:38720)
0928 16:38:59.800110 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:59.800184 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:38:59.800431 18674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.122.253:36151: connect: Connection refused (error 111)
I20260928 16:38:59.801476 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:59.801202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:36151 (local address 127.0.0.1:38736)
0928 16:38:59.801331 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:59.801394 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 16:38:59.801667 18725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.253:36151: Network error: Client connection negotiation failed: client connection to 127.27.122.253:36151: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.253.keytab.
Entry for principal kudu/127.27.122.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.253.keytab.
I20260928 16:38:59.804857 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.253:36151
--webserver_interface=127.27.122.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.253.keytab
--principal=kudu/127.27.122.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:46159,127.27.122.253:36151,127.27.122.252:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:38:59.850906 18725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } attempt: 1
I20260928 16:38:59.851797 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:59.851495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:36151 (local address 127.0.0.1:38748)
0928 16:38:59.851655 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:59.851719 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 16:38:59.852505 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:59.852300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:36151 (local address 127.0.0.1:38764)
0928 16:38:59.852407 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:59.852456 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 16:38:59.852617 18725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.253:36151: Network error: Client connection negotiation failed: client connection to 127.27.122.253:36151: connect: Connection refused (error 111)
W20260928 16:38:59.912683 18732 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:38:59.913015 18732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:38:59.913091 18732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:38:59.913172 18732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:38:59.917110 18732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:38:59.917196 18732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:38:59.917238 18732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:38:59.917271 18732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:38:59.917302 18732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:38:59.920779 18725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } attempt: 2
I20260928 16:38:59.921628 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:59.921292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:36151 (local address 127.0.0.1:38780)
0928 16:38:59.921466 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:59.921538 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 16:38:59.922355 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:59.922128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:36151 (local address 127.0.0.1:38784)
0928 16:38:59.922237 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:59.922290 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.253:36151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 16:38:59.922485 18725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.253:36151: Network error: Client connection negotiation failed: client connection to 127.27.122.253:36151: connect: Connection refused (error 111)
I20260928 16:38:59.922886 18732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:46159,127.27.122.253:36151,127.27.122.252:44325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.253.keytab
--principal=kudu/127.27.122.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.253:36151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.122.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:38:59.923754 18732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:38:59.924635 18732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:38:59.932806 18737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:59.932852 18740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:38:59.932891 18738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:38:59.933145 18732 server_base.cc:1034] running on GCE node
I20260928 16:38:59.933687 18732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:38:59.934671 18732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:38:59.935839 18732 hybrid_clock.cc:648] HybridClock initialized: now 1790613539935820 us; error 27 us; skew 500 ppm
Sep 28 16:38:59 dist-test-slave-g9sh krb5kdc[18644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:38:59.938627 18732 init.cc:377] Logged in from keytab as kudu/127.27.122.253@KRBTEST.COM (short username kudu)
I20260928 16:38:59.939505 18732 webserver.cc:460] Webserver started at http://127.27.122.253:45807/ using document root <none> and password file <none>
I20260928 16:38:59.940053 18732 fs_manager.cc:359] Metadata directory not provided
I20260928 16:38:59.940153 18732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:38:59.940382 18732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:38:59.942473 18732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/data/instance:
uuid: "76a64b47eedb4cd79fca9f4c06af28f6"
format_stamp: "Formatted at 2026-09-28 16:38:59 on dist-test-slave-g9sh"
server_key: "43360a76073575ae166a1a0cf0affaef"
server_key_iv: "137d3eeba57742a748abb0179d027701"
server_key_version: "encryptionkey@0"
I20260928 16:38:59.943073 18732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/wal/instance:
uuid: "76a64b47eedb4cd79fca9f4c06af28f6"
format_stamp: "Formatted at 2026-09-28 16:38:59 on dist-test-slave-g9sh"
server_key: "43360a76073575ae166a1a0cf0affaef"
server_key_iv: "137d3eeba57742a748abb0179d027701"
server_key_version: "encryptionkey@0"
I20260928 16:38:59.946671 18732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 16:38:59.949162 18747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:38:59.950358 18732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 16:38:59.950487 18732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/wal
uuid: "76a64b47eedb4cd79fca9f4c06af28f6"
format_stamp: "Formatted at 2026-09-28 16:38:59 on dist-test-slave-g9sh"
server_key: "43360a76073575ae166a1a0cf0affaef"
server_key_iv: "137d3eeba57742a748abb0179d027701"
server_key_version: "encryptionkey@0"
I20260928 16:38:59.950587 18732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:38:59.963665 18732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:38:59.966835 18732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:38:59.967031 18732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:38:59.973842 18732 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.253:36151
I20260928 16:38:59.974133 18799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.253:36151 every 8 connection(s)
I20260928 16:38:59.974792 18732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/data/info.pb
I20260928 16:38:59.977424 18800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:38:59.980484 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 18732
I20260928 16:38:59.980624 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-1/wal/instance
I20260928 16:38:59.980973 28139 external_mini_cluster.cc:1428] Setting key 691c205c2d1f5f843c403026da85d0c5
I20260928 16:38:59.981292 18800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.122.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.122.252@KRBTEST.COM" created.
Sep 28 16:38:59 dist-test-slave-g9sh krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.253@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:38:59.996954 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:59.983195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.27.122.254:46159)
0928 16:38:59.983328 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 16:38:59.983332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:38:59.984359 (+ 1027us) server_negotiation.cc:407] Connection header received
0928 16:38:59.985404 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:38:59.985408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:38:59.985478 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:38:59.985591 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:38:59.986682 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:59.987549 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:59.988654 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:38:59.989068 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:38:59.989208 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:59.992830 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:38:59.992834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:38:59.992836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:38:59.992860 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:38:59.994677 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:59.995331 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:59.995333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:59.995335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:59.995412 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:38:59.995830 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:38:59.995832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:38:59.995834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:38:59.996001 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:38:59.996114 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 16:38:59.996627 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 16:38:59.996762 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 16:38:59.997694 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:59.983667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:46159 (local address 127.0.0.1:57956)
0928 16:38:59.984185 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:59.984235 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 16:38:59.985185 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:38:59.985733 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:38:59.985739 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:38:59.986066 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:38:59.986521 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:59.986533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:59.987682 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:59.987685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:59.988522 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:38:59.988531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:38:59.989233 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:38:59.989236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:38:59.989373 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:38:59.989967 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:38:59.989984 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:38:59.992656 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:38:59.994823 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:59.994832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:59.994848 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:59.995234 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:59.995508 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:38:59.995514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:38:59.995519 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:38:59.995733 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:38:59.996117 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:38:59.996127 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:38:59.996373 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 16:38:59.996558 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 16:38:59.996876 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":138,"threads_started":1}
I20260928 16:38:59.999264 18800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:38:59 dist-test-slave-g9sh krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.253@KRBTEST.COM for kudu/127.27.122.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.122.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.252.keytab.
Entry for principal kudu/127.27.122.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.252.keytab.
I20260928 16:39:00.008381 28139 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYGQpme/build/debug/bin/kudu
/tmp/dist-test-taskYGQpme/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.252:44325
--webserver_interface=127.27.122.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.252.keytab
--principal=kudu/127.27.122.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.122.212:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.122.254:46159,127.27.122.253:36151,127.27.122.252:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 16:39:00.009152 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:59.999989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.27.122.253:36151)
0928 16:39:00.000325 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.000341 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.000368 (+ 27us) server_negotiation.cc:407] Connection header received
0928 16:39:00.000418 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.000425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.000646 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.000772 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.001168 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.002021 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.002866 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.003266 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.003359 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.005280 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:00.005289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:00.005299 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:00.005321 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:00.007149 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.007754 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.007761 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.007763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.007852 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.008152 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.008154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.008155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.008377 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:00.008509 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:00.008832 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 16:39:00.009009 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":121,"threads_started":1}
I20260928 16:39:00.009179 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:38:59.999799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:36151 (local address 127.0.0.1:38792)
0928 16:38:59.999956 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 16:38:59.999972 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:00.000053 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:00.000788 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:00.000790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:00.000975 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:00.001065 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.001070 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.002170 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.002174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.002753 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.002762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.003387 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.003389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.003460 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.003862 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:00.003873 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:00.005153 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:00.007367 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.007369 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.007371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.007624 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.007953 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.007957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.007960 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.008057 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.008495 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:00.008500 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:00.008614 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 16:39:00.008745 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 16:39:00.008919 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 16:39:00.011296 18800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:39:00.011967 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.011730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55782)
0928 16:39:00.011839 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.011919 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 16:39:00.012154 18750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
I20260928 16:39:00.013281 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.012988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55786)
0928 16:39:00.013133 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.013200 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 16:39:00.013435 18800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:44325: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
I20260928 16:39:00.013619 18725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } attempt: 3
Sep 28 16:39:00 dist-test-slave-g9sh krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for kudu/127.27.122.253@KRBTEST.COM
I20260928 16:39:00.023299 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.014048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:36151 (local address 127.0.0.1:38796)
0928 16:39:00.014166 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.014180 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:00.014260 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:00.014507 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:00.014508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:00.014649 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:00.014748 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.014753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.016548 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.016551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.017525 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.017535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.018235 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.018238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.018316 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.018767 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:00.018778 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:00.020229 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:00.021983 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.021986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.021987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.022175 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.022442 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.022444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.022445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.022552 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.022936 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:00.022938 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:00.023013 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 16:39:00.023093 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 16:39:00.023191 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 16:39:00.023404 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.014117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.27.122.253:36151)
0928 16:39:00.014245 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.014251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.014269 (+ 18us) server_negotiation.cc:407] Connection header received
0928 16:39:00.014337 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.014343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.014395 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.014474 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.014865 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.016432 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.017657 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.018113 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.018247 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.020368 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:00.020371 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:00.020374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:00.020399 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:00.021875 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.022267 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.022270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.022272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.022353 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.022652 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.022655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.022657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.022846 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:00.022938 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:00.023166 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 16:39:00.023272 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 16:39:00.024008 18725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:39:00.024659 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.024425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55790)
0928 16:39:00.024543 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.024592 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 16:39:00.025367 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.025126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55802)
0928 16:39:00.025262 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.025307 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 16:39:00.025520 18725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:44325: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
I20260928 16:39:00.056672 18725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } attempt: 1
I20260928 16:39:00.057513 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.057220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55814)
0928 16:39:00.057372 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.057435 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 16:39:00.058255 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.058032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55828)
0928 16:39:00.058162 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.058205 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 16:39:00.058387 18725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:44325: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
I20260928 16:39:00.062713 18800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } attempt: 1
I20260928 16:39:00.063400 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.063139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55832)
0928 16:39:00.063275 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.063334 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 16:39:00.064163 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.063963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55844)
0928 16:39:00.064062 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.064108 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 16:39:00.064312 18800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:44325: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
W20260928 16:39:00.123569 18806 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 16:39:00.123819 18806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 16:39:00.123842 18806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 16:39:00.123924 18806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 16:39:00.127938 18806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 16:39:00.127990 18806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 16:39:00.128010 18806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 16:39:00.128026 18806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 16:39:00.128042 18806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 16:39:00.132566 18800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } attempt: 2
I20260928 16:39:00.133433 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.133142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55858)
0928 16:39:00.133290 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.133355 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 16:39:00.133402 18806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.122.212:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.122.254:46159,127.27.122.253:36151,127.27.122.252:44325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/krb5kdc/kudu_127.27.122.252.keytab
--principal=kudu/127.27.122.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.122.252:44325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.122.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 15:03:50 UTC on ccycloud.quasar-dswmeh.root.comops.site
build id 4677368
I20260928 16:39:00.133558 18725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } attempt: 2
I20260928 16:39:00.134305 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.133993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55874)
0928 16:39:00.134164 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.134227 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 16:39:00.134645 18806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 16:39:00.134829 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.134573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55890)
0928 16:39:00.134711 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.134767 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 16:39:00.134984 18800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:44325: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
I20260928 16:39:00.135311 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.135102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55892)
0928 16:39:00.135201 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.135252 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 16:39:00.135448 18725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.122.252:44325: Network error: Client connection negotiation failed: client connection to 127.27.122.252:44325: connect: Connection refused (error 111)
I20260928 16:39:00.135885 18806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 16:39:00.144368 18812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 16:39:00.144459 18815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:00.144507 18806 server_base.cc:1034] running on GCE node
W20260928 16:39:00.144457 18813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 16:39:00.145201 18806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 16:39:00.146222 18806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 16:39:00.147413 18806 hybrid_clock.cc:648] HybridClock initialized: now 1790613540147388 us; error 36 us; skew 500 ppm
Sep 28 16:39:00 dist-test-slave-g9sh krb5kdc[18644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613540, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 16:39:00.150138 18806 init.cc:377] Logged in from keytab as kudu/127.27.122.252@KRBTEST.COM (short username kudu)
I20260928 16:39:00.150980 18806 webserver.cc:460] Webserver started at http://127.27.122.252:42385/ using document root <none> and password file <none>
I20260928 16:39:00.151526 18806 fs_manager.cc:359] Metadata directory not provided
I20260928 16:39:00.151620 18806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 16:39:00.151858 18806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 16:39:00.153913 18806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/data/instance:
uuid: "f13d697cb3874b189e4e996ec10aec6f"
format_stamp: "Formatted at 2026-09-28 16:39:00 on dist-test-slave-g9sh"
server_key: "5f8637a05bd19f8ae48fe6be00958749"
server_key_iv: "b32ae3a25aaa59605a3453c8a47f4929"
server_key_version: "encryptionkey@0"
I20260928 16:39:00.154503 18806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/wal/instance:
uuid: "f13d697cb3874b189e4e996ec10aec6f"
format_stamp: "Formatted at 2026-09-28 16:39:00 on dist-test-slave-g9sh"
server_key: "5f8637a05bd19f8ae48fe6be00958749"
server_key_iv: "b32ae3a25aaa59605a3453c8a47f4929"
server_key_version: "encryptionkey@0"
I20260928 16:39:00.158104 18806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 16:39:00.160424 18822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 16:39:00.161760 18806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 16:39:00.161903 18806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/data,/tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/wal
uuid: "f13d697cb3874b189e4e996ec10aec6f"
format_stamp: "Formatted at 2026-09-28 16:39:00 on dist-test-slave-g9sh"
server_key: "5f8637a05bd19f8ae48fe6be00958749"
server_key_iv: "b32ae3a25aaa59605a3453c8a47f4929"
server_key_version: "encryptionkey@0"
I20260928 16:39:00.162015 18806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 16:39:00.192212 18806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 16:39:00.195534 18806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 16:39:00.195751 18806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 16:39:00.202948 18806 rpc_server.cc:307] RPC server started. Bound to: 127.27.122.252:44325
I20260928 16:39:00.203258 18874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.122.252:44325 every 8 connection(s)
I20260928 16:39:00.203913 18806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/data/info.pb
I20260928 16:39:00.205356 28139 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYGQpme/build/debug/bin/kudu as pid 18806
I20260928 16:39:00.205483 28139 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYGQpme/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790613255639823-28139-0/minicluster-data/master-2/wal/instance
I20260928 16:39:00.205794 28139 external_mini_cluster.cc:1428] Setting key 75ac1d8a71fbb5a0cea5cc942abfad63
I20260928 16:39:00.206974 18875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 16:39:00.211556 18875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 16:39:00 dist-test-slave-g9sh krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.252@KRBTEST.COM
Sep 28 16:39:00 dist-test-slave-g9sh krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613540, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.252@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
I20260928 16:39:00.220832 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.207191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.27.122.252:44325)
0928 16:39:00.207599 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.207618 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.207659 (+ 41us) server_negotiation.cc:407] Connection header received
0928 16:39:00.208519 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.208543 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.208983 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.209329 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.210029 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.211490 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.212414 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.213078 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.213273 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.216131 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:00.216136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:00.216144 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:00.216176 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:00.217821 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.218310 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.218318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.218330 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.218431 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.218694 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.218700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.218705 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.219083 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:00.219285 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:00.219568 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 16:39:00.219857 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":107,"threads_started":1}
I20260928 16:39:00.223361 18728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.213364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57970 (local address 127.27.122.254:46159)
0928 16:39:00.213509 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.213514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.213904 (+ 390us) server_negotiation.cc:407] Connection header received
0928 16:39:00.214080 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.214084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.214131 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.214205 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.215089 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.215547 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.216489 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.216871 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.217034 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.219144 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:00.219150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:00.219155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:00.219200 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:00.220866 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.222014 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.222018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.222019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.222110 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.222372 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.222377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.222381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.222644 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:00.222748 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:00.223095 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 16:39:00.223243 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 16:39:00.223932 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.213325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.254:46159 (local address 127.0.0.1:57970)
0928 16:39:00.213757 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.213817 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:00.213955 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:00.214293 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:00.214300 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:00.214790 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:00.214963 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.214973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.215743 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.215746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.216339 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.216348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.217051 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.217056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.217199 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.217677 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:00.217689 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:00.219000 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:00.220990 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.221002 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.221699 (+ 697us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.221905 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.222205 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.222207 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.222209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.222280 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.222747 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:00.222751 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:00.222840 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 16:39:00.223511 (+ 671us) client_negotiation.cc:240] Negotiation successful
0928 16:39:00.223696 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":142,"threads_started":1}
I20260928 16:39:00.225100 18875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:39:00.226148 18800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } attempt: 3
Sep 28 16:39:00 dist-test-slave-g9sh krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.254@KRBTEST.COM
Sep 28 16:39:00 dist-test-slave-g9sh krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613540, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.252@KRBTEST.COM for kudu/127.27.122.253@KRBTEST.COM
Sep 28 16:39:00 dist-test-slave-g9sh krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.253@KRBTEST.COM for kudu/127.27.122.252@KRBTEST.COM
I20260928 16:39:00.234467 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.225548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.253:36151 (local address 127.0.0.1:38810)
0928 16:39:00.225684 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.225698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:00.225781 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:00.226038 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:00.226040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:00.226247 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:00.226363 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.226373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.227333 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.227340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.228298 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.228309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.229010 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.229015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.229115 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.229576 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:00.229587 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:00.230810 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:00.232728 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.232731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.232733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.232995 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.233286 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.233288 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.233289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.233366 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.233938 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:00.233940 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:00.234016 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 16:39:00.234140 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 16:39:00.234320 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 16:39:00.234493 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.225682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38810 (local address 127.27.122.253:36151)
0928 16:39:00.225830 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.225835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.225850 (+ 15us) server_negotiation.cc:407] Connection header received
0928 16:39:00.225892 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.225895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.225936 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.226025 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.226490 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.227216 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.228440 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.228808 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.228968 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.230939 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:00.230942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:00.230945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:00.231009 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:00.232615 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.233100 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.233103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.233105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.233193 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.233472 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.233475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.233478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.233795 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:00.233887 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:00.234225 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 16:39:00.234332 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61,"spinlock_wait_cycles":1792}
I20260928 16:39:00.234588 18725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } attempt: 3
I20260928 16:39:00.235284 18875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } has no permanent_uuid. Determining permanent_uuid...
I20260928 16:39:00.235865 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.223036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.27.122.254:46159)
0928 16:39:00.223381 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.223387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.223398 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:39:00.223424 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.223426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.223485 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.223571 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.224409 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.225165 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.226189 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.226554 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.226654 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.230404 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:00.230411 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:00.230414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:00.230438 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:00.233068 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.233553 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.233558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.233563 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.233651 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.233995 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.233999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.234004 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.234231 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:00.234435 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:00.235552 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0928 16:39:00.235671 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":108,"threads_started":1}
I20260928 16:39:00.237391 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.226534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55914)
0928 16:39:00.226657 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.226671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:00.226739 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:00.227296 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:00.227300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:00.227458 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:00.227531 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.227536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.229047 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.229052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.229721 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.229730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.230363 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.230367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.230459 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.230953 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:00.230984 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:00.232463 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:00.235377 (+ 2914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.235383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.235385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.235645 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.236225 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.236228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.236230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.236326 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.236874 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:00.236877 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:00.237010 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 16:39:00.237098 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 16:39:00.237232 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"spinlock_wait_cycles":4864}
I20260928 16:39:00.237982 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.226751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.27.122.252:44325)
0928 16:39:00.226850 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.226853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.226864 (+ 11us) server_negotiation.cc:407] Connection header received
0928 16:39:00.226894 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.226898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.226940 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.227026 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.227662 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.228207 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.229852 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.230227 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.230340 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.232604 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:00.232608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:00.232610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:00.232634 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:00.235262 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.235769 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.235772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.235774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.235848 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.236553 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.236556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.236557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.236759 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:00.237665 (+ 906us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:00.237743 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 16:39:00.237851 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 16:39:00 dist-test-slave-g9sh krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613539, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.254@KRBTEST.COM for kudu/127.27.122.252@KRBTEST.COM
I20260928 16:39:00.241355 18800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6: Bootstrap starting.
Sep 28 16:39:00 dist-test-slave-g9sh krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.122.253@KRBTEST.COM
I20260928 16:39:00.244374 18800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:00.245527 18729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.234927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55926)
0928 16:39:00.235083 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.235096 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:00.235167 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:00.235903 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:00.235907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:00.236110 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:00.236216 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.236222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.237564 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.237594 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.238356 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.238367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.239008 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.239013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.239094 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.239569 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:00.239585 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:00.240943 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:00.243492 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.243495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.243497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.243737 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.244074 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.244078 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.244080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.244170 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.244993 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:00.244998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:00.245098 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 16:39:00.245227 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 16:39:00.245359 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 16:39:00.245473 18800 log.cc:824] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:00.245555 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.235000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.27.122.252:44325)
0928 16:39:00.235651 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.235664 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.235676 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:39:00.235708 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.235711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.235752 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.236327 (+ 575us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.236442 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.237330 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.238509 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.238869 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.241079 (+ 2210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.241202 (+ 123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:00.241206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:00.241208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:00.241243 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:00.243368 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.243857 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.243861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.243863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.243952 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.244507 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.244513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.244515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.244691 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:00.244810 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:00.245283 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 16:39:00.245390 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":127,"threads_started":1}
I20260928 16:39:00.247351 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.237205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.27.122.253:36151)
0928 16:39:00.237302 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.237306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.237453 (+ 147us) server_negotiation.cc:407] Connection header received
0928 16:39:00.238241 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.238244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.238287 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.238337 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.239161 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.239813 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.240811 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.241143 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.241218 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.244021 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:00.244024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:00.244034 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:00.244057 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:00.245698 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.246192 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.246198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.246203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.246296 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.246592 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.246597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.246598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.246762 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:00.246955 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:00.247108 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 16:39:00.247212 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 16:39:00.248471 18725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a: Bootstrap starting.
I20260928 16:39:00.250445 18800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6: No bootstrap required, opened a new log
I20260928 16:39:00.250748 28139 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 16:39:00.251031 18725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:00.251864 18725 log.cc:824] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a: Log is configured to *not* fsync() on all Append() calls
I20260928 16:39:00.251937 18699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cbc53ea61be4449487012389f0e7359a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57980
Sep 28 16:39:00 dist-test-slave-g9sh krb5kdc[18644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790613540, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.122.252@KRBTEST.COM for kudu/127.27.122.252@KRBTEST.COM
I20260928 16:39:00.254782 18725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a: No bootstrap required, opened a new log
I20260928 16:39:00.256484 18800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } }
I20260928 16:39:00.256719 18800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a64b47eedb4cd79fca9f4c06af28f6, State: Initialized, Role: FOLLOWER
I20260928 16:39:00.257095 18800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } }
I20260928 16:39:00.257436 18725 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } }
I20260928 16:39:00.257653 18725 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc53ea61be4449487012389f0e7359a, State: Initialized, Role: FOLLOWER
I20260928 16:39:00.257634 18885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } } }
I20260928 16:39:00.257789 18885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:39:00.257946 18725 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } }
I20260928 16:39:00.258347 18886 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } } }
I20260928 16:39:00.258466 18886 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:39:00.258586 18800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:00.259012 18725 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:00.259095 18699 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 16:39:00.259232 18699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:00.260550 18699 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } }
I20260928 16:39:00.262221 18774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc53ea61be4449487012389f0e7359a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "76a64b47eedb4cd79fca9f4c06af28f6"
I20260928 16:39:00.262444 18774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:00.262696 18699 leader_election.cc:288] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [CANDIDATE]: Term 1 election: Requested vote from peers 76a64b47eedb4cd79fca9f4c06af28f6 (127.27.122.253:36151), f13d697cb3874b189e4e996ec10aec6f (127.27.122.252:44325)
I20260928 16:39:00.263799 18774 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc53ea61be4449487012389f0e7359a in term 1.
I20260928 16:39:00.263898 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.235716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.122.252:44325 (local address 127.0.0.1:55932)
0928 16:39:00.236842 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0928 16:39:00.236854 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 16:39:00.236975 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 16:39:00.250577 (+ 13602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 16:39:00.250580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 16:39:00.250724 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 16:39:00.250792 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.250824 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.251549 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.251555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.252343 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 16:39:00.252354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.253002 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.253005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 16:39:00.253079 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.253580 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 16:39:00.253595 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 16:39:00.255118 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 16:39:00.258035 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.258038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.258113 (+ 75us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.258570 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.262923 (+ 4353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 16:39:00.262928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 16:39:00.262930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 16:39:00.263021 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 16:39:00.263405 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 16:39:00.263408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 16:39:00.263493 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 16:39:00.263572 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 16:39:00.263724 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074}
I20260928 16:39:00.264474 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.236679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.27.122.252:44325)
0928 16:39:00.246175 (+ 9496us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.246180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.246193 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:39:00.246226 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.246229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.246270 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.249038 (+ 2768us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.250938 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.251444 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.252467 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.252772 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.252864 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.255249 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:00.255253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:00.255256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:00.255277 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:00.257866 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.258797 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.258800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.258801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.258964 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.263118 (+ 4154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.263121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.263123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.263312 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:00.264154 (+ 842us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:00.264225 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 16:39:00.264331 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9269,"thread_start_us":70,"threads_started":1}
W20260928 16:39:00.264885 18902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 16:39:00.265002 18674 leader_election.cc:302] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76a64b47eedb4cd79fca9f4c06af28f6, cbc53ea61be4449487012389f0e7359a; no voters:
W20260928 16:39:00.265156 18902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:39:00.265266 18886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 16:39:00.265489 18886 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [term 1 LEADER]: Becoming Leader. State: Replica: cbc53ea61be4449487012389f0e7359a, State: Running, Role: LEADER
I20260928 16:39:00.265848 18886 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } }
W20260928 16:39:00.266614 18678 tablet.cc:2369] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 16:39:00.267556 18886 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc53ea61be4449487012389f0e7359a. Latest consensus state: current_term: 1 leader_uuid: "cbc53ea61be4449487012389f0e7359a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } } }
I20260928 16:39:00.267658 18886 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:00.267654 18875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f: Bootstrap starting.
I20260928 16:39:00.271193 18875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f: Neither blocks nor log segments found. Creating new log.
I20260928 16:39:00.272006 18875 log.cc:824] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f: Log is configured to *not* fsync() on all Append() calls
W20260928 16:39:00.272760 18908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:39:00.272835 18908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:39:00.275326 18657 master_runner.cc:186] Error getting master registration for 127.27.122.252:44325: OK, instance_id { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" instance_seqno: 1790613540201115 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:39:00.275429 18732 master_runner.cc:186] Error getting master registration for 127.27.122.252:44325: OK, instance_id { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" instance_seqno: 1790613540201115 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 16:39:00.278738 18657 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:39:00.278817 18919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 16:39:00.278863 18732 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 16:39:00.279807 18919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 16:39:00.285914 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.268068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57994 (local address 127.27.122.254:46159)
0928 16:39:00.269065 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.269070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.269082 (+ 12us) server_negotiation.cc:407] Connection header received
0928 16:39:00.269118 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.269121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.269164 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.269239 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.273330 (+ 4091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.274143 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.279794 (+ 5651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.280167 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.280246 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.282270 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:00.282276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:00.282279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:00.282304 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:00.284079 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.284564 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.284568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.284570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.284658 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.284972 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.284976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.284978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.285197 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:00.285307 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:00.285638 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 16:39:00.285761 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":918}
I20260928 16:39:00.296208 18875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f: No bootstrap required, opened a new log
I20260928 16:39:00.300940 18875 raft_consensus.cc:348] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } }
I20260928 16:39:00.301471 18875 raft_consensus.cc:729] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f13d697cb3874b189e4e996ec10aec6f, State: Initialized, Role: FOLLOWER
I20260928 16:39:00.301808 18875 consensus_queue.cc:260] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } }
I20260928 16:39:00.303117 18875 sys_catalog.cc:564] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 16:39:00.303149 18849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cbc53ea61be4449487012389f0e7359a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f13d697cb3874b189e4e996ec10aec6f"
I20260928 16:39:00.303381 18849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f [term 0 FOLLOWER]: Advancing to term 1
I20260928 16:39:00.305174 18849 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc53ea61be4449487012389f0e7359a in term 1.
I20260928 16:39:00.306191 18848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f [term 1 FOLLOWER]: Refusing update from remote peer cbc53ea61be4449487012389f0e7359a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:00.306897 18914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:00.308648 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.269369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38826 (local address 127.27.122.253:36151)
0928 16:39:00.269481 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.269485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.286606 (+ 17121us) server_negotiation.cc:407] Connection header received
0928 16:39:00.286763 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.286766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.286811 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.286857 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.287815 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.288616 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.289738 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.290090 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.290156 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.305067 (+ 14911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:00.305071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:00.305074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:00.305103 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:00.306822 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.307326 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.307330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.307332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.307417 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.307703 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.307707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.307708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.307912 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:00.308045 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:00.308366 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 16:39:00.308486 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 16:39:00.310875 18774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6 [term 1 FOLLOWER]: Refusing update from remote peer cbc53ea61be4449487012389f0e7359a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 16:39:00.311388 18914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 16:39:00.312498 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:00.267903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.27.122.252:44325)
0928 16:39:00.282030 (+ 14127us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.282035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.291907 (+ 9872us) server_negotiation.cc:407] Connection header received
0928 16:39:00.292013 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.292016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.292092 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.292194 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.293500 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.294576 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.296872 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:00.297364 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:00.297463 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:00.300031 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 16:39:00.300035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 16:39:00.300038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 16:39:00.300063 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 16:39:00.302733 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.311186 (+ 8453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.311189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.311191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.311278 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 16:39:00.311617 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 16:39:00.311621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 16:39:00.311622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 16:39:00.311833 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 16:39:00.311934 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:00.312254 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 16:39:00.312362 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14044}
I20260928 16:39:00.315075 18885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc53ea61be4449487012389f0e7359a. Latest consensus state: current_term: 1 leader_uuid: "cbc53ea61be4449487012389f0e7359a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } } }
I20260928 16:39:00.315191 18885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:39:00.316295 18914 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cbc53ea61be4449487012389f0e7359a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } } }
I20260928 16:39:00.316421 18914 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:00.315857 18923 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbc53ea61be4449487012389f0e7359a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } } }
I20260928 16:39:00.318441 18923 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:39:00.318573 18923 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc53ea61be4449487012389f0e7359a. Latest consensus state: current_term: 1 leader_uuid: "cbc53ea61be4449487012389f0e7359a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } } }
I20260928 16:39:00.318670 18923 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:39:00.318755 18923 sys_catalog.cc:455] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cbc53ea61be4449487012389f0e7359a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } } }
I20260928 16:39:00.318825 18923 sys_catalog.cc:458] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f [sys.catalog]: This master's current role is: FOLLOWER
W20260928 16:39:00.321493 18939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 16:39:00.321564 18939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 16:39:00.321969 18922 mvcc.cc:205] Tried to move back new op lower bound from 7334353061044871168 to 7334353060932165632. Current Snapshot: MvccSnapshot[applied={T|T < 7334353061044871168}]
I20260928 16:39:00.323958 18919 catalog_manager.cc:1347] Generated new cluster ID: 134f180e57844c9da0fecee35c82d5bc
I20260928 16:39:00.324015 18919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 16:39:00.325064 18886 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cbc53ea61be4449487012389f0e7359a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } } }
I20260928 16:39:00.325165 18886 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a [sys.catalog]: This master's current role is: LEADER
I20260928 16:39:00.326459 18885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cbc53ea61be4449487012389f0e7359a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc53ea61be4449487012389f0e7359a" member_type: VOTER last_known_addr { host: "127.27.122.254" port: 46159 } } peers { permanent_uuid: "76a64b47eedb4cd79fca9f4c06af28f6" member_type: VOTER last_known_addr { host: "127.27.122.253" port: 36151 } } peers { permanent_uuid: "f13d697cb3874b189e4e996ec10aec6f" member_type: VOTER last_known_addr { host: "127.27.122.252" port: 44325 } } }
I20260928 16:39:00.326567 18885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 16:39:00.327734 18935 mvcc.cc:205] Tried to move back new op lower bound from 7334353061044871168 to 7334353060932165632. Current Snapshot: MvccSnapshot[applied={T|T < 7334353061044871168}]
I20260928 16:39:00.350800 18919 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 16:39:00.352015 18919 catalog_manager.cc:1504] Loading token signing keys...
I20260928 16:39:00.369002 18919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbc53ea61be4449487012389f0e7359a: Generated new TSK 0
I20260928 16:39:00.369906 18919 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 16:39:00.406780 18878 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:00.403660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.27.122.252:44325)
0928 16:39:00.403795 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.403799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.405694 (+ 1895us) server_negotiation.cc:407] Connection header received
0928 16:39:00.405889 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.405892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.405942 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.405997 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.406632 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55942: BlockingRecv error: recv got EOF from 127.0.0.1:55942 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 16:39:00.408536 18880 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:00.404441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58006 (local address 127.27.122.254:46159)
0928 16:39:00.404548 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.404552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.407432 (+ 2880us) server_negotiation.cc:407] Connection header received
0928 16:39:00.407831 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.407834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.407881 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.407931 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.408432 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58006: BlockingRecv error: recv got EOF from 127.0.0.1:58006 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 16:39:00.409804 18805 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:00.405072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.27.122.253:36151)
0928 16:39:00.405184 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.405189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.407631 (+ 2442us) server_negotiation.cc:407] Connection header received
0928 16:39:00.409074 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.409077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.409128 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.409180 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.409693 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38834: BlockingRecv error: recv got EOF from 127.0.0.1:38834 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 16:39:00.414464 18805 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:00.412996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38844 (local address 127.27.122.253:36151)
0928 16:39:00.413104 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.413107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.413538 (+ 431us) server_negotiation.cc:407] Connection header received
0928 16:39:00.413763 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.413766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.413836 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.413888 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.414372 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38844: BlockingRecv error: recv got EOF from 127.0.0.1:38844 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 16:39:00.419668 18805 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:00.418205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38856 (local address 127.27.122.253:36151)
0928 16:39:00.418301 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.418304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.418811 (+ 507us) server_negotiation.cc:407] Connection header received
0928 16:39:00.418984 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.418987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.419036 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.419083 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.419565 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38856: BlockingRecv error: recv got EOF from 127.0.0.1:38856 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260928 16:39:00.425316 18805 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:00.423854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38860 (local address 127.27.122.253:36151)
0928 16:39:00.423948 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.423952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.424436 (+ 484us) server_negotiation.cc:407] Connection header received
0928 16:39:00.424615 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.424618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.424676 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.424726 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.425236 (+ 510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38860: BlockingRecv error: recv got EOF from 127.0.0.1:38860 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 16:39:00.433084 18805 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:00.431537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.27.122.253:36151)
0928 16:39:00.431683 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.431687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.432183 (+ 496us) server_negotiation.cc:407] Connection header received
0928 16:39:00.432375 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.432378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.432423 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.432471 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.432998 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38868: BlockingRecv error: recv got EOF from 127.0.0.1:38868 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 16:39:00.444895 18805 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:00.443434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38874 (local address 127.27.122.253:36151)
0928 16:39:00.443528 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.443532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.444031 (+ 499us) server_negotiation.cc:407] Connection header received
0928 16:39:00.444221 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.444224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.444268 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.444313 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.444805 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38874: BlockingRecv error: recv got EOF from 127.0.0.1:38874 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 16:39:00.464547 18805 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:00.463108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.27.122.253:36151)
0928 16:39:00.463209 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.463213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.463682 (+ 469us) server_negotiation.cc:407] Connection header received
0928 16:39:00.463861 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.463865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.463911 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.463958 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.464431 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38880: BlockingRecv error: recv got EOF from 127.0.0.1:38880 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 16:39:00.500626 18805 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:00.499312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.27.122.253:36151)
0928 16:39:00.499458 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.499464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.499670 (+ 206us) server_negotiation.cc:407] Connection header received
0928 16:39:00.499854 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.499858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.499905 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.500002 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.500480 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38882: BlockingRecv error: recv got EOF from 127.0.0.1:38882 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 16:39:00.568430 18805 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:00.567181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.27.122.253:36151)
0928 16:39:00.567345 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.567350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.567510 (+ 160us) server_negotiation.cc:407] Connection header received
0928 16:39:00.567704 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.567707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.567756 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.567853 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.568322 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38890: BlockingRecv error: recv got EOF from 127.0.0.1:38890 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 16:39:00.700255 18805 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:00.698911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.27.122.253:36151)
0928 16:39:00.699100 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.699105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.699274 (+ 169us) server_negotiation.cc:407] Connection header received
0928 16:39:00.699503 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.699506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.699556 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.699654 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.700147 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38898: BlockingRecv error: recv got EOF from 127.0.0.1:38898 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 16:39:00.960024 18805 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 16:39:00.958809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.27.122.253:36151)
0928 16:39:00.959011 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:00.959017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:00.959034 (+ 17us) server_negotiation.cc:407] Connection header received
0928 16:39:00.959226 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:00.959232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:00.959287 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:00.959381 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 16:39:00.959901 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38904: BlockingRecv error: recv got EOF from 127.0.0.1:38904 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 16:39:01.269634 18902 catalog_manager.cc:1259] Loaded cluster ID: 134f180e57844c9da0fecee35c82d5bc
I20260928 16:39:01.269742 18902 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6: loading cluster ID for follower catalog manager: success
I20260928 16:39:01.271443 18902 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6: acquiring CA information for follower catalog manager: success
I20260928 16:39:01.272593 18902 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 76a64b47eedb4cd79fca9f4c06af28f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:39:01.322783 18939 catalog_manager.cc:1259] Loaded cluster ID: 134f180e57844c9da0fecee35c82d5bc
I20260928 16:39:01.322861 18939 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f: loading cluster ID for follower catalog manager: success
I20260928 16:39:01.324811 18939 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f: acquiring CA information for follower catalog manager: success
I20260928 16:39:01.326534 18939 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f13d697cb3874b189e4e996ec10aec6f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 16:39:01.477900 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:01.474183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.27.122.253:36151)
0928 16:39:01.474429 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:01.474433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:01.474466 (+ 33us) server_negotiation.cc:407] Connection header received
0928 16:39:01.474606 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:01.474609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:01.474735 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:01.474832 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:01.475078 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:01.475617 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:01.476417 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:01.476661 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:01.476740 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:01.477057 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:01.477505 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:01.477602 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:01.477655 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 16:39:01.477701 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":68,"threads_started":1}
I20260928 16:39:01.484489 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:01.480794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55952 (local address 127.27.122.252:44325)
0928 16:39:01.481075 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:01.481078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:01.481091 (+ 13us) server_negotiation.cc:407] Connection header received
0928 16:39:01.481218 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:01.481220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:01.481329 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:01.481421 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:01.481674 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:01.482160 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:01.482997 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:01.483302 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:01.483381 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:01.483697 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:01.484095 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:01.484190 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:01.484262 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 16:39:01.484305 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
I20260928 16:39:01.491488 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:01.488109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.27.122.252:44325)
0928 16:39:01.488203 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:01.488206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:01.488575 (+ 369us) server_negotiation.cc:407] Connection header received
0928 16:39:01.488701 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:01.488704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:01.488745 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:01.488829 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:01.489085 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:01.489547 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:01.490405 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:01.490672 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:01.490770 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:01.490990 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:01.491069 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:01.491132 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:01.491310 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 16:39:01.491357 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 16:39:01.491848 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:01.487934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.27.122.253:36151)
0928 16:39:01.488059 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:01.488063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:01.488077 (+ 14us) server_negotiation.cc:407] Connection header received
0928 16:39:01.488235 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:01.488241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:01.488291 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:01.488335 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:01.488656 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:01.489379 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:01.490683 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:01.491011 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:01.491126 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:01.491405 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:01.491485 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:01.491530 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:01.491685 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 16:39:01.491733 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 16:39:01.492465 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 16:39:01.487456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.27.122.254:46159)
0928 16:39:01.487727 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 16:39:01.487734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 16:39:01.487770 (+ 36us) server_negotiation.cc:407] Connection header received
0928 16:39:01.487862 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 16:39:01.487867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 16:39:01.488024 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 16:39:01.488149 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 16:39:01.488486 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:01.489314 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:01.490692 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 16:39:01.491034 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 16:39:01.491163 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 16:39:01.491572 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 16:39:01.491873 (+ 301us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 16:39:01.491976 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 16:39:01.492235 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 16:39:01.492297 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
I20260928 16:39:01.495609 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 18657
I20260928 16:39:01.502894 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 18732
I20260928 16:39:01.509449 28139 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYGQpme/build/debug/bin/kudu with pid 18806
2026-09-28T16:39:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3093 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6186 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285862 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST